[jira] [Commented] (HIVE-22437) LLAP Metadata cache NPE on locking metadata.
[
https://issues.apache.org/jira/browse/HIVE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021818#comment-17021818
]
Hive QA commented on HIVE-22437:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} llap-server: The patch generated 3 new + 27 unchanged
- 2 fixed = 30 total (was 29) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} llap-server generated 0 new + 88 unchanged - 2 fixed
= 88 total (was 90) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20291/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20291/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20291/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP Metadata cache NPE on locking metadata.
>
>
> Key: HIVE-22437
> URL: https://issues.apache.org/jira/browse/HIVE-22437
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22437.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.unlockSingleBuffer(MetadataCache.java:464)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockBuffer(MetadataCache.java:409)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockOldVal(MetadataCache.java:314)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putInternal(MetadataCache.java:287)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:199)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021815#comment-17021815
]
Hive QA commented on HIVE-22746:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991553/HIVE-22746.01.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17945 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=227)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991553 - PreCommit-HIVE-Build
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22746.01.patch, HIVE-22746.patch
>
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021794#comment-17021794
]
Hive QA commented on HIVE-22746:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 142 new + 1331 unchanged - 15
fixed = 1473 total (was 1346) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 1 new + 1530 unchanged - 2 fixed = 1531
total (was 1532) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
59s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of procCtx at line 1499 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node,
Object) At TypeCheckProcFactory.java:1499 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.type.TypeCheckProcFactory.processGByExpr(Node,
Object) At TypeCheckProcFactory.java:[line 1492] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20290/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20290/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20290/yetus/new-findbugs-ql.html
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20290/yetus/diff-javadoc-javadoc-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20290/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22746.01.patch, HIVE-22746.patch
>
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021785#comment-17021785
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991552/HIVE-22736.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20289/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20289/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12991552/HIVE-22736.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991552 - PreCommit-HIVE-Build
> Support replication across multiple encryption zones
>
>
> Key: HIVE-22736
> URL: https://issues.apache.org/jira/browse/HIVE-22736
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22736.patch, HIVE-22736.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers
[
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021784#comment-17021784
]
Hive QA commented on HIVE-22653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991551/HIVE-22653.08.patch
{color:green}SUCCESS:{color} +1 due to 26 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20288/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20288/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991551 - PreCommit-HIVE-Build
> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
> Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch,
> HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch,
> HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch,
> HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch,
> HIVE-22653.08.patch, HIVE-22653.08.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct
> dependency, however a high number of files still refer to commons-lang, which
> is transitively brought in either way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers
[
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021748#comment-17021748
]
Hive QA commented on HIVE-22653:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 17s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-20288/patches/PreCommit-HIVE-Build-20288.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20288/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
> Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch,
> HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch,
> HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch,
> HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch,
> HIVE-22653.08.patch, HIVE-22653.08.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct
> dependency, however a high number of files still refer to commons-lang, which
> is transitively brought in either way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[
https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021742#comment-17021742
]
Hive QA commented on HIVE-22731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991544/HIVE-22731.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20287/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20287/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:50:44.414
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20287/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:50:44.416
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:50:45.092
+ rm -rf ../yetus_PreCommit-HIVE-Build-20287
+ mkdir ../yetus_PreCommit-HIVE-Build-20287
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20287
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20287/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java:
does not exist in index
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java:
does not exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java:19
Falling back to three-way merge...
[jira] [Commented] (HIVE-22641) Columns returned in sorted order when show columns query is run with no search pattern.
[
https://issues.apache.org/jira/browse/HIVE-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021736#comment-17021736
]
Hive QA commented on HIVE-22641:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991534/HIVE-22641.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20286/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20286/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20286/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:49:29.177
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20286/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:49:29.180
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 03:49:30.338
+ rm -rf ../yetus_PreCommit-HIVE-Build-20286
+ mkdir ../yetus_PreCommit-HIVE-Build-20286
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20286
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20286/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
Trying to apply the patch with -p1
error: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist
in index
Trying to apply the patch with -p2
error: src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in
index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-20286
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991534 - PreCommit-HIVE-Build
> Columns returned in sorted order when show columns query is run with no
> search pattern.
> ---
>
> Key: HIVE-22641
> URL: https://issues.apache.org/jira/browse/HIVE-22641
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Attachments: HIVE-22641.patch
>
>
> In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
> in same order as it was created. for example
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+--+
> | field |
> +-+--+
> | server_name |
> | task_name |
> | partition_name |
> | start_time |
> | end_time|
> | table_owner |
> | table_name |
> +-+--+
> {code}
> For Hive 3 columns are returned in sorted order for the same query, below is
> output.
> {code}
> create table
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[
https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021734#comment-17021734
]
Hive QA commented on HIVE-22753:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991520/HIVE-22753.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20285/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991520 - PreCommit-HIVE-Build
> Fix gradual mem leak: Operationlog related appenders should be cleared up on
> errors
>
>
> Key: HIVE-22753
> URL: https://issues.apache.org/jira/browse/HIVE-22753
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch,
> image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png,
> image-2020-01-21-11-18-37-294.png
>
>
> In case of exception in SQLOperation, operational log does not get cleared
> up. This causes gradual build up of HushableRandomAccessFileAppender causing
> HS2 to OOM after some time.
> !image-2020-01-21-11-14-37-911.png|width=431,height=267!
>
> Allocation tree
> !image-2020-01-21-11-18-37-294.png|width=425,height=178!
>
> Prod instance mem
> !image-2020-01-21-11-17-59-279.png|width=698,height=209!
>
> Each HushableRandomAccessFileAppender holds internal ref to
> RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem
> leak.
> Related ticket: HIVE-18820
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[
https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021711#comment-17021711
]
Hive QA commented on HIVE-22753:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20285/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20285/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix gradual mem leak: Operationlog related appenders should be cleared up on
> errors
>
>
> Key: HIVE-22753
> URL: https://issues.apache.org/jira/browse/HIVE-22753
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch,
> image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png,
> image-2020-01-21-11-18-37-294.png
>
>
> In case of exception in SQLOperation, operational log does not get cleared
> up. This causes gradual build up of HushableRandomAccessFileAppender causing
> HS2 to OOM after some time.
> !image-2020-01-21-11-14-37-911.png|width=431,height=267!
>
> Allocation tree
> !image-2020-01-21-11-18-37-294.png|width=425,height=178!
>
> Prod instance mem
> !image-2020-01-21-11-17-59-279.png|width=698,height=209!
>
> Each HushableRandomAccessFileAppender holds internal ref to
> RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem
> leak.
> Related ticket: HIVE-18820
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Attachment: HIVE-22366.6.patch
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, HIVE-22366.6.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Patch Available (was: Open)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, HIVE-22366.6.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22366:
---
Status: Open (was: Patch Available)
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch, HIVE-22366.6.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[
https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021688#comment-17021688
]
Hive QA commented on HIVE-22518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991522/HIVE-22518.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20284/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991522 - PreCommit-HIVE-Build
> SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
>
>
> Key: HIVE-22518
> URL: https://issues.apache.org/jira/browse/HIVE-22518
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch,
> HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch,
> HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch,
> HIVE-22518.05.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22318) Java.io.exception:Two readers for
[
https://issues.apache.org/jira/browse/HIVE-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021685#comment-17021685
]
Bernard commented on HIVE-22318:
I've observed that this bug occurs only when the destination table isn't
partitioned or bucketed. Can someone confirm?
> Java.io.exception:Two readers for
> -
>
> Key: HIVE-22318
> URL: https://issues.apache.org/jira/browse/HIVE-22318
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: max_c
>Priority: Major
> Attachments: hiveserver2 for exception.log
>
>
> I create a ACID table with ORC format:
>
> {noformat}
> CREATE TABLE `some.TableA`(
>
>)
> ROW FORMAT SERDE
>'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
>'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> TBLPROPERTIES (
>'bucketing_version'='2',
>'orc.compress'='SNAPPY',
>'transactional'='true',
>'transactional_properties'='default'){noformat}
> After executing merge into operation:
> {noformat}
> MERGE INTO some.TableA AS a USING (SELECT vend_no FROM some.TableB UNION ALL
> SELECT vend_no FROM some.TableC) AS b ON a.vend_no=b.vend_no WHEN MATCHED
> THEN DELETE
> {noformat}
> the problem happend(when selecting the TableA, the exception happens too):
> {noformat}
> java.io.IOException: java.io.IOException: Two readers for {originalWriteId:
> 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}: new
> [key={originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId
> 25}, nextRecord={2, 4, 536870912, 2434, 25, null}, reader=Hive ORC
> Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_1,
> 9223372036854775807)], old [key={originalWriteId: 4, bucket:
> 536870912(1.0.0), row: 2434, currentWriteId 25}, nextRecord={2, 4, 536870912,
> 2434, 25, null}, reader=Hive ORC
> Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_0{noformat}
> Through orc_tools I scan all the
> files(bucket_0,bucket_1,bucket_2) under delete_delta and find all
> rows of files are the same.I think this will cause the same
> key(RecordIdentifer) when scan the bucket_1 after bucket_0 but I
> don't know why all the rows are the same in these bucket files.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[
https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021676#comment-17021676
]
Hive QA commented on HIVE-22518:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20284/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql . itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20284/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
>
>
> Key: HIVE-22518
> URL: https://issues.apache.org/jira/browse/HIVE-22518
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch,
> HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch,
> HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch,
> HIVE-22518.05.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021649#comment-17021649
]
Hive QA commented on HIVE-22758:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991499/HIVE-22758.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20283/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20283/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20283/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-23 00:51:12.558
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20283/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 00:51:12.561
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-23 00:51:13.475
+ rm -rf ../yetus_PreCommit-HIVE-Build-20283
+ mkdir ../yetus_PreCommit-HIVE-Build-20283
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20283
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20283/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
fatal: corrupt patch at line 42
Trying to apply the patch with -p1
fatal: corrupt patch at line 42
Trying to apply the patch with -p2
fatal: corrupt patch at line 42
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-20283
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991499 - PreCommit-HIVE-Build
> Create database with permission error when doas set to true
> ---
>
> Key: HIVE-22758
> URL: https://issues.apache.org/jira/browse/HIVE-22758
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Critical
> Attachments: HIVE-22758.1.patch
>
>
> With doAs set to true, running create database on external location fails
> with permission denied for write access on the directory for hive user (User
> HMS is running as).
> Steps to reproduce the issue:
> 1. Turn on, Hive run as end-user to true.
> 2. Connect to hive as some user other than admin, eg:- chiran
> 3. Create a database with external location
> {code}
> create database externaldbexample location '/user/chiran/externaldbexample'
> {code}
> The above statement fails as write access is not available to hive service
> user on HDFS as below.
> {code}
> > create database externaldbexample location '/user/chiran/externaldbexample';
> INFO : Compiling
> command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
> create database externaldbexample location '/user/chiran/externaldbexample'
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021648#comment-17021648
]
Hive QA commented on HIVE-22629:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991472/HIVE-22629.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991472 - PreCommit-HIVE-Build
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch,
> HIVE-22629.3.patch, HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22629) AST Node Children can be quite expensive to build due to List resizing
[
https://issues.apache.org/jira/browse/HIVE-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021631#comment-17021631
]
Hive QA commented on HIVE-22629:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 376 unchanged - 1
fixed = 376 total (was 377) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20282/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20282/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> AST Node Children can be quite expensive to build due to List resizing
> --
>
> Key: HIVE-22629
> URL: https://issues.apache.org/jira/browse/HIVE-22629
> Project: Hive
> Issue Type: Improvement
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22629.1.patch, HIVE-22629.2.patch,
> HIVE-22629.3.patch, HIVE-22629.patch,
> noETLs_ETLs_profile-kc-hdp-mstr06-p.servicemanagement.com-interactive-166620-t-e-cpu-1576029590.svg
>
>
> As per the attached profile, The AST Node can be a major source of CPU and
> memory churn, due to the ArrayList resizing and copy.
> In my Opinion this can be amortized by providing the actual size.
> [~jcamachorodriguez] / [~vgarg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021617#comment-17021617
]
Hive QA commented on HIVE-22736:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991552/HIVE-22736.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 469 failed/errored test(s), 15842 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=227)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestListPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestLockRequestBuilder - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestMetaStoreEndFunctionListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestMetaStoreEventListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestMetaStoreEventListenerWithOldConf - did not produce a TEST-*.xml file
(likely timed out) (batchId=238)
TestMetastoreConf - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestMetastoreScheduledQueries - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestMetrics - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestNotNullConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestObjectStoreSchemaMethods - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestObjectStoreStatementVerify - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestPartitionManagement - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
(likely timed out) (batchId=229)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestPrimaryKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestRawStoreProxy - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestRemoteHiveMetaStoreZKBindHost - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestRuntimeStats - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestStats - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestTablesCreateDropAlterTruncate - did not produce a TEST-*.xml file (likely
timed out) (batchId=227)
TestTablesList - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestTxnHandlerNegative - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
TestTxnUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=227)
[jira] [Commented] (HIVE-22736) Support replication across multiple encryption zones
[
https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021588#comment-17021588
]
Hive QA commented on HIVE-22736:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} shims/common: The patch generated 2 new + 94 unchanged
- 0 fixed = 96 total (was 94) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 26 new + 98 unchanged - 0 fixed = 124 total (was 98) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 450 unchanged - 0 fixed = 451 total (was 450) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 3 new + 255 unchanged - 0
fixed = 258 total (was 255) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 15 new + 0
unchanged - 0 fixed = 15 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 58s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Updated] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22761: Attachment: HIVE-22761.01.patch > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22761.01.patch > > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22761: Status: Patch Available (was: Open) > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22761.01.patch > > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22761) Scheduled query executor fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22761: Summary: Scheduled query executor fails to report query state as errored if session initialization fails (was: Scheduled query execution fails to report query state as errored if session initialization fails) > Scheduled query executor fails to report query state as errored if session > initialization fails > --- > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22761) Scheduled query should be reported as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22761: Summary: Scheduled query should be reported as errored if session initialization fails (was: Scheduled query should be reported as errored in session initialization fails) > Scheduled query should be reported as errored if session initialization fails > - > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22761) Scheduled query execution fails to report query state as errored if session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22761: Summary: Scheduled query execution fails to report query state as errored if session initialization fails (was: Scheduled query should be reported as errored if session initialization fails) > Scheduled query execution fails to report query state as errored if session > initialization fails > > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22761) Scheduled query should be reported as errored in session initialization fails
[ https://issues.apache.org/jira/browse/HIVE-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22761: --- > Scheduled query should be reported as errored in session initialization fails > - > > Key: HIVE-22761 > URL: https://issues.apache.org/jira/browse/HIVE-22761 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > right now the info object is only initialized after the sessionstate is > inited - which might get into trouble... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Patch Available (was: Open) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: HIVE-22726.3.patch > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch, > HIVE-22726.3.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Open (was: Patch Available) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[
https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021517#comment-17021517
]
Hive QA commented on HIVE-20801:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976705/HIVE-20801.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:28:52.830
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20280/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:28:52.833
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:28:53.513
+ rm -rf ../yetus_PreCommit-HIVE-Build-20280
+ mkdir ../yetus_PreCommit-HIVE-Build-20280
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20280
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20280/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:2561
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:2338
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java'
cleanly.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:11: trailing whitespace.
"unlocked partitition by partition, instead of being transactionally
scoped.\n" +
/data/hiveptest/working/scratch/build.patch:70: trailing whitespace.
// Test non-strict read-locking mode - the read locks are only obtained for
the ACID side
error: patch failed:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:2561
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:2338
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java'
cleanly.
U common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
warning: 2 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-20280
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976705 - PreCommit-HIVE-Build
> ACID: Allow DbTxnManager to ignore non-ACID table locking
> -
>
> Key: HIVE-20801
> URL: https://issues.apache.org/jira/browse/HIVE-20801
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 4.0.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>
[jira] [Commented] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[
https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021515#comment-17021515
]
Hive QA commented on HIVE-22726:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991518/HIVE-22726.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20279/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20279/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:27:37.143
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20279/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:27:37.146
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 05cabc8 HIVE-22666: Introduce TopNKey operator for PTF Reduce
Sink (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 21:27:37.871
+ rm -rf ../yetus_PreCommit-HIVE-Build-20279
+ mkdir ../yetus_PreCommit-HIVE-Build-20279
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20279
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20279/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperBatch.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneralComparator.java:
does not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java: does not
exist in index
Trying to apply the patch with -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java:38
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java' with
conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java:58
Falling back to three-way merge...
Applied patch to
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[
https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021514#comment-17021514
]
Hive QA commented on HIVE-22627:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991523/HIVE-22627.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20278/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20278/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991523 - PreCommit-HIVE-Build
> Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
> -
>
> Key: HIVE-22627
> URL: https://issues.apache.org/jira/browse/HIVE-22627
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch,
> HIVE-22627.4.patch, HIVE-22627.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[
https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021458#comment-17021458
]
Hive QA commented on HIVE-22627:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20278/dev-support/hive-personality.sh
|
| git revision | master / 05cabc8 |
| modules | C: metastore U: metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20278/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
> -
>
> Key: HIVE-22627
> URL: https://issues.apache.org/jira/browse/HIVE-22627
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Chovan
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch,
> HIVE-22627.4.patch, HIVE-22627.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021447#comment-17021447
]
Hive QA commented on HIVE-22366:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991490/HIVE-22366.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17940 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
(batchId=229)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
(batchId=229)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
(batchId=229)
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20277/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20277/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20277/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991490 - PreCommit-HIVE-Build
> Multiple metastore calls for same table and constraints during planning
> ---
>
> Key: HIVE-22366
> URL: https://issues.apache.org/jira/browse/HIVE-22366
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22366.1.patch, HIVE-22366.2.patch,
> HIVE-22366.3.patch, HIVE-22366.4.patch, HIVE-22366.5.patch
>
>
> e.g. for a simple query like
> {code:sql}
> explain select count(i) from test2 group by j limit 3;
> {code}
> planner makes the following calls:
> {code}
> 2019-10-17T22:44:49,892 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:49,908 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:49,910 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:49,911 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,913 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:49,979 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_partitions : tbl=hive.default.test2
> 2019-10-17T22:44:49,997 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_aggr_stats_for: table=hive.default.test2
> 2019-10-17T22:44:50,014 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr cmd=get_table :
> tbl=hive.default.test2
> 2019-10-17T22:44:50,023 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_primary_keys : tbl=hive.default.test2
> 2019-10-17T22:44:50,024 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_foreign_keys : parentdb=null parenttbl=null foreigndb=default
> foreigntbl=test2
> 2019-10-17T22:44:50,025 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_unique_constraints : tbl=hive.default.test2
> 2019-10-17T22:44:50,026 INFO [812910d4-bc6a-450c-aa8b-3487b4edceab main]
> HiveMetaStore.audit: ugi=vgarg ip=unknown-ip-addr
> cmd=get_not_null_constraints : tbl=hive.default.test2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22578) CBO genOPTree is not failsafe for CTAS and VIEW statements
[ https://issues.apache.org/jira/browse/HIVE-22578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021435#comment-17021435 ] Jesus Camacho Rodriguez commented on HIVE-22578: [~hamvas.aron], [~vgarg], this patch seems to introduce a call to create a duplicate of the complete AST tree, which is a very expensive operation. Is there an alternative fix that would avoid that call? > CBO genOPTree is not failsafe for CTAS and VIEW statements > -- > > Key: HIVE-22578 > URL: https://issues.apache.org/jira/browse/HIVE-22578 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 3.0.0, 3.1.2 >Reporter: Aron Hamvas >Assignee: Aron Hamvas >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22578.2.patch, HIVE-22578.patch > > > If CBO fails during OT generation, it is supposed to skip CBO and fall back > to SemanticAnalyzer generating the OT for the original AST. However, for CTAS > and VIEW statements, the original AST is discarded and replaced with the new > AST. > If OT generation for the new AST fails with a SemanticException, the > compilation is destined to fail completely since we cannot fall back to the > original and possibly valid AST. > Also, the SemanticException sometimes gets hidden and misjudged as a missing > column statistics issue due to bad error handling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22366) Multiple metastore calls for same table and constraints during planning
[
https://issues.apache.org/jira/browse/HIVE-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021408#comment-17021408
]
Hive QA commented on HIVE-22366:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
15s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
8s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 3 new + 570 unchanged - 37
fixed = 573 total (was 607) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
21s{color} | {color:red} ql generated 6 new + 1532 unchanged - 0 fixed = 1538
total (was 1532) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field org.apache.hadoop.hive.ql.metadata.Table.cc is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
| | The field org.apache.hadoop.hive.ql.metadata.Table.dc is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
| | The field org.apache.hadoop.hive.ql.metadata.Table.fki is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
| | The field org.apache.hadoop.hive.ql.metadata.Table.nnc is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
| | The field org.apache.hadoop.hive.ql.metadata.Table.pki is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
| | The field org.apache.hadoop.hive.ql.metadata.Table.uki is transient but
isn't set by deserialization In Table.java:but isn't set by deserialization
In Table.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20277/dev-support/hive-personality.sh
|
| git revision | master / a7ca0a7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20277/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20277/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-common ql U: .
[jira] [Updated] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22666:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~kkasa]!
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.5.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde:
[jira] [Commented] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021371#comment-17021371
]
Hive QA commented on HIVE-22666:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991503/HIVE-22666.5.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17945 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20276/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20276/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20276/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991503 - PreCommit-HIVE-Build
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.5.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> tag: -1
> TopN: 4
> TopN Hash Memory Usage: 0.1
> auto parallelism: true
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> Path -> Alias:
> A masked pattern was here
> Path -> Partition:
> A masked pattern was here
> Partition
> base file name: testtable_n1000
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
> column.name.delimiter ,
> columns s_state,ss_net_profit
> columns.comments
> columns.types string:double
> A masked pattern was here
> name default.testtable_n1000
> numFiles 1
> numRows 10
> rawDataSize 80
> serialization.ddl struct testtable_n1000 { string
> s_state, double ss_net_profit}
> serialization.format 1
> serialization.lib
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> totalSize 90
> A masked pattern was here
> serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> properties:
> COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"s_state":"true","ss_net_profit":"true"}}
> bucket_count -1
> bucketing_version 2
>
[jira] [Assigned] (HIVE-22760) Add Clock caching eviction based strategy
[ https://issues.apache.org/jira/browse/HIVE-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slim Bouguerra reassigned HIVE-22760: - > Add Clock caching eviction based strategy > - > > Key: HIVE-22760 > URL: https://issues.apache.org/jira/browse/HIVE-22760 > Project: Hive > Issue Type: New Feature > Components: llap >Reporter: Slim Bouguerra >Assignee: Slim Bouguerra >Priority: Major > > LRFU is the current default right now. > The main issue with such Strategy is that it has a very high memory overhead, > in addition to that, most of the accounting has to happen under locks thus > can be source of contentions. > Add Simpler policy like clock, can help with both issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22437) LLAP Metadata cache NPE on locking metadata.
[
https://issues.apache.org/jira/browse/HIVE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22437:
--
Attachment: HIVE-22437.patch
> LLAP Metadata cache NPE on locking metadata.
>
>
> Key: HIVE-22437
> URL: https://issues.apache.org/jira/browse/HIVE-22437
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
> Attachments: HIVE-22437.patch
>
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.unlockSingleBuffer(MetadataCache.java:464)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockBuffer(MetadataCache.java:409)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockOldVal(MetadataCache.java:314)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putInternal(MetadataCache.java:287)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:199)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22437) LLAP Metadata cache NPE on locking metadata.
[
https://issues.apache.org/jira/browse/HIVE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22437:
--
Status: Patch Available (was: Open)
> LLAP Metadata cache NPE on locking metadata.
>
>
> Key: HIVE-22437
> URL: https://issues.apache.org/jira/browse/HIVE-22437
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.unlockSingleBuffer(MetadataCache.java:464)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockBuffer(MetadataCache.java:409)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockOldVal(MetadataCache.java:314)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putInternal(MetadataCache.java:287)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:199)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22437) LLAP Metadata cache NPE on locking metadata.
[
https://issues.apache.org/jira/browse/HIVE-22437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Slim Bouguerra updated HIVE-22437:
--
Component/s: llap
> LLAP Metadata cache NPE on locking metadata.
>
>
> Key: HIVE-22437
> URL: https://issues.apache.org/jira/browse/HIVE-22437
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Slim Bouguerra
>Assignee: Slim Bouguerra
>Priority: Major
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.unlockSingleBuffer(MetadataCache.java:464)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockBuffer(MetadataCache.java:409)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.lockOldVal(MetadataCache.java:314)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putInternal(MetadataCache.java:287)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:199)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22666) Introduce TopNKey operator for PTF Reduce Sink
[
https://issues.apache.org/jira/browse/HIVE-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021305#comment-17021305
]
Hive QA commented on HIVE-22666:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20276/dev-support/hive-personality.sh
|
| git revision | master / a7ca0a7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20276/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce TopNKey operator for PTF Reduce Sink
> --
>
> Key: HIVE-22666
> URL: https://issues.apache.org/jira/browse/HIVE-22666
> Project: Hive
> Issue Type: Improvement
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22666.1.patch, HIVE-22666.2.patch,
> HIVE-22666.3.patch, HIVE-22666.3.patch, HIVE-22666.4.patch,
> HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.4.patch, HIVE-22666.5.patch
>
>
> {code}
> EXPLAIN EXTENDED
> SELECT s_state, ranking
> FROM (
> SELECT s_state AS s_state,
> rank() OVER (PARTITION BY s_state ORDER BY ss_net_profit) AS ranking
> FROM testtable_n1000) tmp1
> WHERE ranking <= 3;
> {code}
> {code}
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Tez
> A masked pattern was here
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: testtable_n1000
> Statistics: Num rows: 10 Data size: 940 Basic stats:
> COMPLETE Column stats: COMPLETE
> GatherStats: false
> Reduce Output Operator
> key expressions: s_state (type: string), ss_net_profit
> (type: double)
> null sort order: az
> sort order: ++
> Map-reduce partition columns: s_state (type: string)
> Statistics: Num rows: 10 Data size:
[jira] [Updated] (HIVE-22746) Make TypeCheckProcFactory generic
[
https://issues.apache.org/jira/browse/HIVE-22746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22746:
---
Attachment: HIVE-22746.01.patch
> Make TypeCheckProcFactory generic
> -
>
> Key: HIVE-22746
> URL: https://issues.apache.org/jira/browse/HIVE-22746
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-22746.01.patch, HIVE-22746.patch
>
>
> {{TypeCheckProcFactory}} is responsible for processing AST nodes and
> generating ExprNode objects from them. When we generate the expressions for
> Calcite planning, we go through a {{AST node -> ExprNode -> RexNode}}
> transformation. We would like to avoid the overhead of going through the
> ExprNode, and thus generate directly the RexNode from the AST.
> To do that, the first step is to make {{TypeCheckProcFactory}} generic, so it
> can receive an expression factory and create expressions in different realms.
> For the time being, the only factory implementation is the ExprNode factory.
> Thus, this patch focuses mainly on refactoring {{TypeCheckProcFactory}}
> without breaking anything that is already working.
> In a follow-up patch, we will create a {{RexNode}} factory and use it when we
> parse the query in CalcitePlanner.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Status: In Progress (was: Patch Available) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22736: --- Attachment: HIVE-22736.patch Status: Patch Available (was: In Progress) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch, HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers
[ https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22653: Attachment: HIVE-22653.08.patch > Remove commons-lang leftovers > - > > Key: HIVE-22653 > URL: https://issues.apache.org/jira/browse/HIVE-22653 > Project: Hive > Issue Type: Bug >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, > HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch, > HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch, > HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch, > HIVE-22653.08.patch, HIVE-22653.08.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct > dependency, however a high number of files still refer to commons-lang, which > is transitively brought in either way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers
[
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021276#comment-17021276
]
Hive QA commented on HIVE-22653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991511/HIVE-22653.08.patch
{color:green}SUCCESS:{color} +1 due to 26 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 17071 tests
executed
*Failed tests:*
{noformat}
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestMetaStoreEndFunctionListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestMetastoreConf - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestMetastoreScheduledQueries - did not produce a TEST-*.xml file (likely timed
out) (batchId=229)
TestMetrics - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestNotNullConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestObjectStoreStatementVerify - did not produce a TEST-*.xml file (likely
timed out) (batchId=229)
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
(likely timed out) (batchId=229)
TestPrimaryKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
TestStats - did not produce a TEST-*.xml file (likely timed out) (batchId=229)
TestTablesGetExists - did not produce a TEST-*.xml file (likely timed out)
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20275/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20275/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20275/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991511 - PreCommit-HIVE-Build
> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
> Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch,
> HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch,
> HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch,
> HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch,
> HIVE-22653.08.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct
> dependency, however a high number of files still refer to commons-lang, which
> is transitively brought in either way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Status: Patch Available (was: In Progress) > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22731.1.patch, HIVE-22731.2.patch, > HIVE-22731.WIP.patch, decode_time_bars.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Status: In Progress (was: Patch Available) > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22731.1.patch, HIVE-22731.2.patch, > HIVE-22731.WIP.patch, decode_time_bars.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[ https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-22731: -- Attachment: HIVE-22731.2.patch > Probe MapJoin hashtables for row level filtering > > > Key: HIVE-22731 > URL: https://issues.apache.org/jira/browse/HIVE-22731 > Project: Hive > Issue Type: Improvement > Components: Hive, llap >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22731.1.patch, HIVE-22731.2.patch, > HIVE-22731.WIP.patch, decode_time_bars.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, RecordReaders such as ORC support filtering at coarser-grained > levels, namely: File, Stripe (64 to 256mb), and Row group (10k row) level. > They only filter sets of rows if they can guarantee that none of the rows can > pass a filter (usually given as searchable argument). > However, a significant amount of time can be spend decoding rows with > multiple columns that are not even used in the final result. See figure where > original is what happens today and in LazyDecode we skip decoding rows that > do not match the key. > To enable a more fine-grained filtering in the particular case of a MapJoin > we could utilize the key HashTable created from the smaller table to skip > deserializing row columns at the larger table that do not match any key and > thus save CPU time. > This Jira investigates this direction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reopened HIVE-22126: -- Assignee: Eugene Chung (was: Laszlo Pinter) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021223#comment-17021223 ] Laszlo Pinter commented on HIVE-22126: -- [~euigeun_chung] feel free to reopen and take over the jira. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22653) Remove commons-lang leftovers
[
https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021203#comment-17021203
]
Hive QA commented on HIVE-22653:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 18s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-20275/patches/PreCommit-HIVE-Build-20275.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20275/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove commons-lang leftovers
> -
>
> Key: HIVE-22653
> URL: https://issues.apache.org/jira/browse/HIVE-22653
> Project: Hive
> Issue Type: Bug
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch,
> HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch,
> HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch,
> HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch,
> HIVE-22653.08.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct
> dependency, however a high number of files still refer to commons-lang, which
> is transitively brought in either way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22731) Probe MapJoin hashtables for row level filtering
[
https://issues.apache.org/jira/browse/HIVE-22731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021199#comment-17021199
]
Hive QA commented on HIVE-22731:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991394/HIVE-22731.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20274/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20274/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20274/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-01-22 15:28:22.744
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-20274/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 15:28:22.747
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a7ca0a7 HIVE-22663: Quote all table and column names or do not
quote any (Zoltan Chovan reviewed by Peter Vary)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a7ca0a7 HIVE-22663: Quote all table and column names or do not
quote any (Zoltan Chovan reviewed by Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-01-22 15:28:24.210
+ rm -rf ../yetus_PreCommit-HIVE-Build-20274
+ mkdir ../yetus_PreCommit-HIVE-Build-20274
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-20274
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-20274/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java:19
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java:
patch does not apply
Trying to apply the patch with -p1
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java:
does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java: does
not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java:
does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java:
does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java:
does not exist in index
error: src/test/resources/testconfiguration.properties: does not exist in index
error:
src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java: does
not exist in index
error:
src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java:
does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java:
does not exist in index
error:
src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java:
does not exist in index
error:
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021195#comment-17021195
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991508/HIVE-22729.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20273/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20273/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20273/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12991508/HIVE-22729.06.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991508 - PreCommit-HIVE-Build
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[
https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021194#comment-17021194
]
Hive QA commented on HIVE-22756:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991482/HIVE-22756.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17939 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20272/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20272/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20272/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991482 - PreCommit-HIVE-Build
> Drop XAttrs options for distcp when dest is a blobstore
> ---
>
> Key: HIVE-22756
> URL: https://issues.apache.org/jira/browse/HIVE-22756
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22756.1.patch
>
>
> Drop XAttrs options for distcp when dest is a blobstore
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22758) Create database with permission error when doas set to true
[
https://issues.apache.org/jira/browse/HIVE-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22758:
-
Description:
With doAs set to true, running create database on external location fails with
permission denied for write access on the directory for hive user (User HMS is
running as).
Steps to reproduce the issue:
1. Turn on, Hive run as end-user to true.
2. Connect to hive as some user other than admin, eg:- chiran
3. Create a database with external location
{code}
create database externaldbexample location '/user/chiran/externaldbexample'
{code}
The above statement fails as write access is not available to hive service user
on HDFS as below.
{code}
> create database externaldbexample location '/user/chiran/externaldbexample';
INFO : Compiling
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
create database externaldbexample location '/user/chiran/externaldbexample'
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d); Time
taken: 1.377 seconds
INFO : Executing
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d):
create database externaldbexample location '/user/chiran/externaldbexample'
INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.reflect.UndeclaredThrowableException)
INFO : Completed executing
command(queryId=hive_20200122043626_5c95e1fd-ce00-45fd-b58d-54f5e579f87d); Time
taken: 0.238 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLTask.
MetaException(message:java.lang.reflect.UndeclaredThrowableException)
(state=08S01,code=1)
{code}
>From Hive Metastore service log, below is seen.
{code}
2020-01-22T04:36:27,870 WARN [pool-6-thread-6]: metastore.ObjectStore
(ObjectStore.java:getDatabase(1010)) - Failed to get database
hive.externaldbexample, returning NoSuchObjectExcept
ion
2020-01-22T04:36:27,898 INFO [pool-6-thread-6]: metastore.HiveMetaStore
(HiveMetaStore.java:run(1339)) - Creating database path in managed directory
hdfs://c470-node2.squadron.support.
hortonworks.com:8020/user/chiran/externaldbexample
2020-01-22T04:36:27,903 INFO [pool-6-thread-6]: utils.FileUtils
(FileUtils.java:mkdir(170)) - Creating directory if it doesn't exist:
hdfs://namenodeaddress:8020/user/chiran/externaldbexample
2020-01-22T04:36:27,932 ERROR [pool-6-thread-6]: utils.MetaStoreUtils
(MetaStoreUtils.java:logAndThrowMetaException(169)) - Got exception:
org.apache.hadoop.security.AccessControlException Permission denied: user=hive,
access=WRITE, inode="/user/chiran":chiran:chiran:drwxr-xr-x
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255)
at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1859)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1843)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1802)
at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3150)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
{code}
The behavior looks to be a regression of HIVE-20001.
Can we add a check to see if the external path specified by a user, then create
DB
[jira] [Updated] (HIVE-22641) Columns returned in sorted order when show columns query is run with no search pattern.
[
https://issues.apache.org/jira/browse/HIVE-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22641:
-
Description:
In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
in same order as it was created. for example
{code}
create table col_order_test(server_name string, task_name string,
partition_name string, start_time string, end_time string, table_owner string,
table_name string) stored as orc;
show columns in col_order_test;
+-+--+
| field |
+-+--+
| server_name |
| task_name |
| partition_name |
| start_time |
| end_time|
| table_owner |
| table_name |
+-+--+
{code}
For Hive 3 columns are returned in sorted order for the same query, below is
output.
{code}
create table col_order_test(server_name string, task_name string,
partition_name string, start_time string, end_time string, table_owner string,
table_name string) stored as orc;
show columns in col_order_test;
+-+
| field |
+-+
| end_time|
| partition_name |
| server_name |
| start_time |
| table_name |
| table_owner |
| task_name |
+-+
{code}
Above behavior looks to be changed with the introduction of search column
feature as part of Jira [HIVE-18373
|https://issues.apache.org/jira/browse/HIVE-18373]
This behavior change can cause an existing process to fail in a few
environments, for example, code to generate the INSERT OVERWRITE in a different
manner, which may result in query failure.
I would like to request a community if we can improve the Jira [HIVE-18373
|https://issues.apache.org/jira/browse/HIVE-18373] by returning column order
same as it was created if the search pattern provided by the user is null.
Attaching patch with the change.
was:
In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
in same order as it was created. for example
{code}
create table col_order_test(server_name string, task_name string,
partition_name string, start_time string, end_time string, table_owner string,
table_name string) stored as orc;
show columns in col_order_test;
+-+--+
| field |
+-+--+
| server_name |
| task_name |
| partition_name |
| start_time |
| end_time|
| table_owner |
| table_name |
+-+--+
{code}
For Hive 3 columns are returned in sorted order for the same query, below is
output.
{code}
create table col_order_test(server_name string, task_name string,
partition_name string, start_time string, end_time string, table_owner string,
table_name string) stored as orc;
show columns in col_order_test;
+-+
| field |
+-+
| end_time|
| partition_name |
| server_name |
| start_time |
| table_name |
| table_owner |
| task_name |
+-+
{code}
Above behavior looks to be changed with the introduction of search column
feature as part of Jira [HIVE-18373
|https://issues.apache.org/jira/browse/HIVE-18373]
This behavior change can cause existing process to fail setup in a few
environments, for example: code to generate the INSERT OVERWRITE in a different
manner, which may result in query failure.
I would like to request a community if we can improve the Jira [HIVE-18373
|https://issues.apache.org/jira/browse/HIVE-18373] by returning column order
same as it was created if the search pattern provided by the user is null.
Attaching patch with the change.
> Columns returned in sorted order when show columns query is run with no
> search pattern.
> ---
>
> Key: HIVE-22641
> URL: https://issues.apache.org/jira/browse/HIVE-22641
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Attachments: HIVE-22641.patch
>
>
> In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
> in same order as it was created. for example
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+--+
> | field |
> +-+--+
> | server_name |
> | task_name |
> | partition_name |
> | start_time |
> | end_time|
> | table_owner |
> | table_name |
> +-+--+
> {code}
> For Hive 3 columns are returned in sorted order for the same query, below is
> output.
> {code}
>
[jira] [Updated] (HIVE-22641) Columns returned in sorted order when show columns query is run with no search pattern.
[
https://issues.apache.org/jira/browse/HIVE-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22641:
-
Attachment: HIVE-22641.patch
Status: Patch Available (was: Open)
> Columns returned in sorted order when show columns query is run with no
> search pattern.
> ---
>
> Key: HIVE-22641
> URL: https://issues.apache.org/jira/browse/HIVE-22641
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Attachments: HIVE-22641.patch
>
>
> In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
> in same order as it was created. for example
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+--+
> | field |
> +-+--+
> | server_name |
> | task_name |
> | partition_name |
> | start_time |
> | end_time|
> | table_owner |
> | table_name |
> +-+--+
> {code}
> For Hive 3 columns are returned in sorted order for the same query, below is
> output.
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+
> | field |
> +-+
> | end_time|
> | partition_name |
> | server_name |
> | start_time |
> | table_name |
> | table_owner |
> | task_name |
> +-+
> {code}
> Above behavior looks to be changed with the introduction of search column
> feature as part of Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373]
> This behavior change can cause existing process to fail setup in a few
> environments, for example: code to generate the INSERT OVERWRITE in a
> different manner, which may result in query failure.
> I would like to request a community if we can improve the Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373] by returning column order
> same as it was created if the search pattern provided by the user is null.
> Attaching patch with the change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22641) Columns returned in sorted order when show columns query is run with no search pattern.
[
https://issues.apache.org/jira/browse/HIVE-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani reassigned HIVE-22641:
Assignee: Chiran Ravani
> Columns returned in sorted order when show columns query is run with no
> search pattern.
> ---
>
> Key: HIVE-22641
> URL: https://issues.apache.org/jira/browse/HIVE-22641
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: Chiran Ravani
>Assignee: Chiran Ravani
>Priority: Major
> Attachments: HIVE-22641.patch
>
>
> In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
> in same order as it was created. for example
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+--+
> | field |
> +-+--+
> | server_name |
> | task_name |
> | partition_name |
> | start_time |
> | end_time|
> | table_owner |
> | table_name |
> +-+--+
> {code}
> For Hive 3 columns are returned in sorted order for the same query, below is
> output.
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+
> | field |
> +-+
> | end_time|
> | partition_name |
> | server_name |
> | start_time |
> | table_name |
> | table_owner |
> | task_name |
> +-+
> {code}
> Above behavior looks to be changed with the introduction of search column
> feature as part of Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373]
> This behavior change can cause existing process to fail setup in a few
> environments, for example: code to generate the INSERT OVERWRITE in a
> different manner, which may result in query failure.
> I would like to request a community if we can improve the Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373] by returning column order
> same as it was created if the search pattern provided by the user is null.
> Attaching patch with the change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22641) Columns returned in sorted order when show columns query is run with no search pattern.
[
https://issues.apache.org/jira/browse/HIVE-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chiran Ravani updated HIVE-22641:
-
Attachment: (was: HIVE-22641.patch)
> Columns returned in sorted order when show columns query is run with no
> search pattern.
> ---
>
> Key: HIVE-22641
> URL: https://issues.apache.org/jira/browse/HIVE-22641
> Project: Hive
> Issue Type: Improvement
> Components: Hive, HiveServer2
>Affects Versions: 3.1.0
>Reporter: Chiran Ravani
>Priority: Major
> Attachments: HIVE-22641.patch
>
>
> In Hive 1.2.1 and 2.0 while displaying columns for a table, it used to return
> in same order as it was created. for example
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+--+
> | field |
> +-+--+
> | server_name |
> | task_name |
> | partition_name |
> | start_time |
> | end_time|
> | table_owner |
> | table_name |
> +-+--+
> {code}
> For Hive 3 columns are returned in sorted order for the same query, below is
> output.
> {code}
> create table col_order_test(server_name string, task_name string,
> partition_name string, start_time string, end_time string, table_owner
> string, table_name string) stored as orc;
> show columns in col_order_test;
> +-+
> | field |
> +-+
> | end_time|
> | partition_name |
> | server_name |
> | start_time |
> | table_name |
> | table_owner |
> | task_name |
> +-+
> {code}
> Above behavior looks to be changed with the introduction of search column
> feature as part of Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373]
> This behavior change can cause existing process to fail setup in a few
> environments, for example: code to generate the INSERT OVERWRITE in a
> different manner, which may result in query failure.
> I would like to request a community if we can improve the Jira [HIVE-18373
> |https://issues.apache.org/jira/browse/HIVE-18373] by returning column order
> same as it was created if the search pattern provided by the user is null.
> Attaching patch with the change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020982#comment-17020982 ] Eugene Chung edited comment on HIVE-22126 at 1/22/20 2:26 PM: -- [^HIVE-22126.01.patch] is incorrect and this issue should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] Plus, some calcite API usages with guava must be adjusted. (RelOptRule#operand()) https://github.com/EugeneChung/hive/commit/19a4e2dd3655702606876a95878150d495955090 was (Author: euigeun_chung): [^HIVE-22126.01.patch] is incorrect and this issue should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-22126:
Comment: was deleted
(was: I tried to attach the file but was denied.
{code:java}
$ cat HIVE-22126.02.patch
diff --git a/ql/pom.xml b/ql/pom.xml
index e5eed46947..016f8d2588 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -1023,8 +1023,12 @@
org.apache.hive.com.zaxxer.hikari
- com.google.guava
-
org.apache.hive.com.google.guava
+ com.google.common
+
org.apache.hive.com.google.common
+
+
+ com.google.thirdparty.publicsuffix
+
org.apache.hive.com.google.thirdparty.publicsuffix
{code}
)
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22756) Drop XAttrs options for distcp when dest is a blobstore
[
https://issues.apache.org/jira/browse/HIVE-22756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021103#comment-17021103
]
Hive QA commented on HIVE-22756:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
17s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} shims/common: The patch generated 1 new + 93 unchanged
- 1 fixed = 94 total (was 94) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} shims/0.23: The patch generated 3 new + 63 unchanged -
2 fixed = 66 total (was 65) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20272/dev-support/hive-personality.sh
|
| git revision | master / a7ca0a7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20272/yetus/diff-checkstyle-shims_common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20272/yetus/diff-checkstyle-shims_0.23.txt
|
| modules | C: shims/common shims/0.23 common U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20272/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Drop XAttrs options for distcp when dest is a blobstore
> ---
>
> Key: HIVE-22756
> URL: https://issues.apache.org/jira/browse/HIVE-22756
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22756.1.patch
>
>
> Drop XAttrs options for distcp when dest is a blobstore
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-22627: - Attachment: HIVE-22627.4.patch > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, > HIVE-22627.4.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22518) SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests
[ https://issues.apache.org/jira/browse/HIVE-22518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22518: Attachment: HIVE-22518.05.patch > SQLStdHiveAuthorizerFactoryForTest doesn't work correctly for llap tests > > > Key: HIVE-22518 > URL: https://issues.apache.org/jira/browse/HIVE-22518 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22518.01.patch, HIVE-22518.01.patch, > HIVE-22518.02.patch, HIVE-22518.02.patch, HIVE-22518.03.patch, > HIVE-22518.03.patch, HIVE-22518.04.patch, HIVE-22518.05.patch, > HIVE-22518.05.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22736) Support replication across multiple encryption zones
[ https://issues.apache.org/jira/browse/HIVE-22736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-22736: -- Assignee: Aasha Medhi (was: mahesh kumar behera) > Support replication across multiple encryption zones > > > Key: HIVE-22736 > URL: https://issues.apache.org/jira/browse/HIVE-22736 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22736.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-22126: Comment: was deleted (was: I was wondering why this patch wasn't applied to 3.x. Is there any specific reason?) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Status: Patch Available (was: Open) > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png, > image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021041#comment-17021041 ] Rajesh Balamohan edited comment on HIVE-22753 at 1/22/20 1:16 PM: -- There is a race between BatchEventProcessor and the cleanup operation performed by HS2 thread. So even though stop() is being invoked and file is getting closed, same filename is recreated by BatchEventProcessor in a span of 1 ms due to race. This instance never gets cleared up, causing the leak. This also creates stale directories/files in ops log folder. Another option with .2 patch is to track the files which are genuinely getting closed and prevent them from getting recreated within seconds. Verified that this fixes the leak in the cluster. was (Author: rajesh.balamohan): There is a race between BatchEventProcessor and the cleanup operation performed by HS2 thread. So even though stop() is being invoked and file is getting closed, same filename is recreated by BatchEventProcessor in a span of 1 ms due to race. This instance never gets cleared up, causing the leak. This also creates stale directories/files in ops log folder. Another option with .2 patch is to track the files which are genuinely getting closed and prevent them from getting recreated within seconds. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png, > image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021041#comment-17021041 ] Rajesh Balamohan commented on HIVE-22753: - There is a race between BatchEventProcessor and the cleanup operation performed by HS2 thread. So even though stop() is being invoked and file is getting closed, same filename is recreated by BatchEventProcessor in a span of 1 ms due to race. This instance never gets cleared up, causing the leak. This also creates stale directories/files in ops log folder. Another option with .2 patch is to track the files which are genuinely getting closed and prevent them from getting recreated within seconds. > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png, > image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22753) Fix gradual mem leak: Operationlog related appenders should be cleared up on errors
[ https://issues.apache.org/jira/browse/HIVE-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-22753: Attachment: HIVE-22753.2.patch > Fix gradual mem leak: Operationlog related appenders should be cleared up on > errors > > > Key: HIVE-22753 > URL: https://issues.apache.org/jira/browse/HIVE-22753 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22753.1.patch, HIVE-22753.2.patch, > image-2020-01-21-11-14-37-911.png, image-2020-01-21-11-17-59-279.png, > image-2020-01-21-11-18-37-294.png > > > In case of exception in SQLOperation, operational log does not get cleared > up. This causes gradual build up of HushableRandomAccessFileAppender causing > HS2 to OOM after some time. > !image-2020-01-21-11-14-37-911.png|width=431,height=267! > > Allocation tree > !image-2020-01-21-11-18-37-294.png|width=425,height=178! > > Prod instance mem > !image-2020-01-21-11-17-59-279.png|width=698,height=209! > > Each HushableRandomAccessFileAppender holds internal ref to > RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem > leak. > Related ticket: HIVE-18820 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22663) Quote all table and column names or do not quote any
[
https://issues.apache.org/jira/browse/HIVE-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22663:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~zchovan]!
> Quote all table and column names or do not quote any
>
>
> Key: HIVE-22663
> URL: https://issues.apache.org/jira/browse/HIVE-22663
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Ashutosh Bapat
>Assignee: Zoltan Chovan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22663.2.patch, HIVE-22663.3.patch,
> HIVE-22663.4.patch, HIVE-22663.5.patch, HIVE-22663.6.patch, HIVE-22663.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The change in HIVE-22546 is causing following stack trace when I run Hive
> with PostgreSQL as backend db for the metastore.
> 0: jdbc:hive2://localhost:1> create database dumpdb with
> ('repl.source.for'='1,2,3');0: jdbc:hive2://localhost:1> create database
> dumpdb with ('repl.source.for'='1,2,3');Error: Error while compiling
> statement: FAILED: ParseException line 1:28 missing KW_DBPROPERTIES at '('
> near '' (state=42000,code=4)0: jdbc:hive2://localhost:1> create
> database dumpdb with dbproperties ('repl.source.for'='1,2,3');ERROR : FAILED:
> Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Error
> communicating with the
> metastore)org.apache.hadoop.hive.ql.lockmgr.LockException: Error
> communicating with the metastore at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.commitTxn(DbTxnManager.java:541)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:687)
> at
> org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:653)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:969)
> ... stack trace clipped
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> MetaException(message:Unable to update transaction database
> org.postgresql.util.PSQLException: ERROR: relation
> "materialization_rebuild_locks" does not exist Position: 13 at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at
> This happens because the table names in all the queries in TxnHandler.java
> (including the one at 1312, which causes this stack trace) are not quoting
> the table names. All the tablenames and column names should be quoted there.
> Just the change in HIVE-22546 won't suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22627) Add schema changes introduced in HIVE-21443 to the schema upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021002#comment-17021002 ] Peter Vary commented on HIVE-22627: --- Can you please check if rebase is needed? > Add schema changes introduced in HIVE-21443 to the schema upgrade scripts > - > > Key: HIVE-22627 > URL: https://issues.apache.org/jira/browse/HIVE-22627 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22627.2.patch, HIVE-22627.3.patch, HIVE-22627.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: (was: HIVE-22726.2.patch) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Open (was: Patch Available) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: HIVE-22726.2.patch > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Status: Patch Available (was: Open) > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22726) TopN Key optimizer should use array instead of priority queue
[ https://issues.apache.org/jira/browse/HIVE-22726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22726: - Attachment: HIVE-22726.2.patch > TopN Key optimizer should use array instead of priority queue > - > > Key: HIVE-22726 > URL: https://issues.apache.org/jira/browse/HIVE-22726 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22726.1.patch, HIVE-22726.2.patch > > > The TopN key optimizer currently uses a priority queue for keeping track of > the largest/smallest rows. Its max size is the same as the user specified > limit. This should be replaced a more cache line friendly array with a small > (128) maximum size and see how much performance is gained. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020996#comment-17020996
]
Hive QA commented on HIVE-22729:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12991508/HIVE-22729.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17940 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.templeton.TestConcurrentJobRequestsThreadsAndTimeout.ConcurrentListJobsVerifyExceptions
(batchId=212)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20271/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20271/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20271/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12991508 - PreCommit-HIVE-Build
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020982#comment-17020982 ] Eugene Chung edited comment on HIVE-22126 at 1/22/20 11:56 AM: --- [^HIVE-22126.01.patch] is incorrect and this issue should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] was (Author: euigeun_chung): [^HIVE-22126.01.patch] is incorrect and should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020987#comment-17020987
]
Eugene Chung edited comment on HIVE-22126 at 1/22/20 11:55 AM:
---
I tried to attach the file but was denied.
{code:java}
$ cat HIVE-22126.02.patch
diff --git a/ql/pom.xml b/ql/pom.xml
index e5eed46947..016f8d2588 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -1023,8 +1023,12 @@
org.apache.hive.com.zaxxer.hikari
- com.google.guava
-
org.apache.hive.com.google.guava
+ com.google.common
+
org.apache.hive.com.google.common
+
+
+ com.google.thirdparty.publicsuffix
+
org.apache.hive.com.google.thirdparty.publicsuffix
{code}
was (Author: euigeun_chung):
I tried to attach the file but was denied.
$ cat HIVE-22126.02.patch
diff --git a/ql/pom.xml b/ql/pom.xml
index e5eed46947..016f8d2588 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -1023,8 +1023,12 @@
org.apache.hive.com.zaxxer.hikari
- com.google.guava
-
org.apache.hive.com.google.guava
+ com.google.common
+
org.apache.hive.com.google.common
+
+
+ com.google.thirdparty.publicsuffix
+
org.apache.hive.com.google.thirdparty.publicsuffix
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020987#comment-17020987 ] Eugene Chung commented on HIVE-22126: - I tried to attach the file but was denied. $ cat HIVE-22126.02.patch diff --git a/ql/pom.xml b/ql/pom.xml index e5eed46947..016f8d2588 100644 --- a/ql/pom.xml +++ b/ql/pom.xml @@ -1023,8 +1023,12 @@ org.apache.hive.com.zaxxer.hikari - com.google.guava - org.apache.hive.com.google.guava + com.google.common + org.apache.hive.com.google.common + + + com.google.thirdparty.publicsuffix + org.apache.hive.com.google.thirdparty.publicsuffix > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020982#comment-17020982 ] Eugene Chung edited comment on HIVE-22126 at 1/22/20 11:52 AM: --- [^HIVE-22126.01.patch] is incorrect and should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] was (Author: euigeun_chung): This is incorrect and should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020982#comment-17020982 ] Eugene Chung commented on HIVE-22126: - This is incorrect and should be reopened. Google guava has two packages starting with com.google.common and com.google.thirdparty.publicsuffix. [https://github.com/google/guava/tree/master/guava/src/com/google] > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020971#comment-17020971
]
Hive QA commented on HIVE-22729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
181 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1532 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 676 unchanged - 7 fixed = 677 total (was 683) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20271/dev-support/hive-personality.sh
|
| git revision | master / 1266f82 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20271/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20271/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20271/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW
[jira] [Updated] (HIVE-22759) Default decimal round off producing incorrect results in "Vectorized" mode
[
https://issues.apache.org/jira/browse/HIVE-22759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh Maheshwari updated HIVE-22759:
Description:
Premise:
External table on top of a parquet file. The table contains decimal with a
fixed scale, and there could be entries in parquet with higher scale. Hive, by
default rounds off the values to the scale defined in table schema and uses a
view of translated values while applying filters.
Steps to reproduce -
a. Create a new HDFS path and [upload attached parquet
files|https://drive.google.com/file/d/1sUg-ovWku9X4AmmwQND2hzg5yIKVTy9q/view?usp=sharing]
within the folder.
b. Create an external table managing these parquet files
{noformat}
create table lineitem_15_3(l_orderkey int, l_suppkey int, l_linenumber int,
l_quantity decimal(15,3), l_extendedprice decimal(15,3), l_discount
decimal(15,3), l_tax decimal(15,3), l_returnflag string, l_linestatus string,
l_shipdate date, l_commitdate date, l_receiptdate date, l_shipinstruct string,
l_shipmode string, l_comment string) stored as parquet location
'hdfs://';
{noformat}
c. Execute following query in vectorized and non-vectorized mode -
{noformat}
select * from lineitem_15_3 where l_extendedprice <= 918.031 and l_discount >=
0.0 and l_tax>= 0.001 and l_orderkey = 107695104;
{noformat}
Non vectorized mode produces correct result (1 row) -
{noformat}
hive> set hive.vectorized.execution.enabled = false;
hive> select * from lineitem_15_3 where l_extendedprice <= 918.031 and
l_discount >= 0.0 and l_tax >= 0.001 and l_orderkey = 107695104; Query ID =
hive_20200122101124_3734cfab-6cbb-4562-97ee-71e15775babe Total jobs = 1
Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no
reduce operator Starting Job = job_1579669891475_0048, Tracking URL = <...>
Kill Command = /opt/hadoop/bin/mapred job -kill job_1579669891475_0048 Hadoop
job information for Stage-1: number of mappers: 2; number of reducers: 0
2020-01-22 10:11:30,661 Stage-1 map = 0%, reduce = 0% 2020-01-22 10:11:39,833
Stage-1 map = 50%, reduce = 0%, Cumulative CPU 7.45 sec 2020-01-22
10:11:46,961 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 25.22 sec
MapReduce Total cumulative CPU time: 25 seconds 220 msec Ended Job =
job_1579669891475_0048 MapReduce Jobs Launched: Stage-Stage-1: Map: 2
Cumulative CPU: 25.22 sec HDFS Read: 318375885 HDFS Write: 310 SUCCESS Total
MapReduce CPU Time Spent: 25 seconds 220 msec
OK
107695104 263020 6 1.000 918.026 0.006 0.001 N O 1995-06-21 1995-08-08
1995-07-14 TAKE BACK RETURN FOB thely express reques
Time taken: 23.24 seconds, Fetched: 1 row(s)
{noformat}
Vectorized mode produces incorrect result (no rows) -
{noformat}
hive> set hive.vectorized.execution.enabled = true;
hive> select * from lineitem_15_3 where l_extendedprice <= 918.031 and
l_discount >= 0.0 and l_tax >= 0.001 and l_orderkey = 107695104;
Query ID = hive_20200122101058_3cda8d56-0885-4c31-aff4-6686a84f639e
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1579669891475_0047, Tracking URL = <...>
Kill Command = /opt/hadoop/bin/mapred job -kill job_1579669891475_0047
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2020-01-22 10:11:04,474 Stage-1 map = 0%, reduce = 0%
2020-01-22 10:11:12,651 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 5.29 sec
2020-01-22 10:11:16,735 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 20.6
sec
MapReduce Total cumulative CPU time: 20 seconds 600 msec
Ended Job = job_1579669891475_0047
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2 Cumulative CPU: 20.6 sec HDFS Read: 318378365 HDFS
Write: 174 SUCCESS
Total MapReduce CPU Time Spent: 20 seconds 600 msec
OK
Time taken: 19.822 seconds{noformat}
Similar behaviour is observed for many other queries. The record in parquet
file has a value of 0.00075 for l_tax. Since, schema defines 3 digits of scale
in decimal points, it should match with 0.001 after rounding off and return the
row.
If we explicitly apply round() function to scale of 3 on l_tax column, the row
shows up correctly.
{noformat}
hive> select * from lineitem_15_3 where l_extendedprice <= 918.031 and
l_discount >= 0.0 and round(l_tax,3) >= 0.001 and l_orderkey = 107695104; Query
ID = hive_20200122103950_c5d579b8-770f-441d-bfaa-74c84ac3c515 Total jobs = 1
Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no
reduce operator Starting Job = job_1579669891475_0049, Tracking URL = <...>
Kill Command = /opt/hadoop/bin/mapred job -kill job_1579669891475_0049 Hadoop
job information for Stage-1: number of mappers: 2; number of reducers: 0
2020-01-22 10:39:56,751 Stage-1 map = 0%, reduce = 0% 2020-01-22 10:40:03,894
Stage-1 map = 50%, reduce = 0%, Cumulative CPU 8.63 sec 2020-01-22
10:40:12,044 Stage-1 map = 100%, reduce = 0%,
[jira] [Updated] (HIVE-22653) Remove commons-lang leftovers
[ https://issues.apache.org/jira/browse/HIVE-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22653: Attachment: HIVE-22653.08.patch > Remove commons-lang leftovers > - > > Key: HIVE-22653 > URL: https://issues.apache.org/jira/browse/HIVE-22653 > Project: Hive > Issue Type: Bug >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, > HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch, > HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch, > HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch, > HIVE-22653.08.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct > dependency, however a high number of files still refer to commons-lang, which > is transitively brought in either way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22653) Remove commons-lang leftovers
[ https://issues.apache.org/jira/browse/HIVE-22653?focusedWorklogId=375516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375516 ] ASF GitHub Bot logged work on HIVE-22653: - Author: ASF GitHub Bot Created on: 22/Jan/20 10:49 Start Date: 22/Jan/20 10:49 Worklog Time Spent: 10m Work Description: dlavati commented on pull request #886: HIVE-22653 commons-lang leftovers URL: https://github.com/apache/hive/pull/886 Since #870 was closed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 375516) Time Spent: 0.5h (was: 20m) > Remove commons-lang leftovers > - > > Key: HIVE-22653 > URL: https://issues.apache.org/jira/browse/HIVE-22653 > Project: Hive > Issue Type: Bug >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22653.01.patch, HIVE-22653.01.patch, > HIVE-22653.02.patch, HIVE-22653.03.patch, HIVE-22653.04.patch, > HIVE-22653.04.patch, HIVE-22653.04.patch, HIVE-22653.05.patch, > HIVE-22653.06.patch, HIVE-22653.07.patch, HIVE-22653.07.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > HIVE-7145 removed commons-lang - in favor of commons-lang3 - as a direct > dependency, however a high number of files still refer to commons-lang, which > is transitively brought in either way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[
https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22729:
-
Attachment: HIVE-22729.06.patch
> Provide a failure reason for failed compactions
> ---
>
> Key: HIVE-22729
> URL: https://issues.apache.org/jira/browse/HIVE-22729
> Project: Hive
> Issue Type: Improvement
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch,
> HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch,
> HIVE-22729.06.patch
>
>
> We should provide a compaction failure reason as easily accessible as
> possible. Like in the result of the {{SHOW COMPACTIONS}} command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020942#comment-17020942 ] Peter Vary commented on HIVE-22755: --- In my deleted comment I misunderstood the purpose of the jira :( The read lock is needed so there is no concurrent DDL command running parallel the compaction. Before having versioned table/partition metadata, removing this read lock could be problematic. > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > > The minOpenTxnId is used by the Cleaner here > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L154 > This currently converts it to open write-ids to clean appropriately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22755) Cleaner/Compaction can skip the read locks and use the min open txn id
[ https://issues.apache.org/jira/browse/HIVE-22755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22755: -- Comment: was deleted (was: I have 2 ideas: * Time based approach: use the start time of the minOpenTxn to filter the files to clean * Instead of skipping every read lock, create a read lock per table instead of per partition. This would fix some failures created by the previous patch with the added benefit of knowing what tables are under read. CC: [~lpinter], [~dkuzmenko]) > Cleaner/Compaction can skip the read locks and use the min open txn id > -- > > Key: HIVE-22755 > URL: https://issues.apache.org/jira/browse/HIVE-22755 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Slim Bouguerra >Priority: Major > Fix For: 4.0.0 > > > The minOpenTxnId is used by the Cleaner here > https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java#L154 > This currently converts it to open write-ids to clean appropriately. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22705) LLAP cache is polluted by query-based compactor
[ https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22705: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks for the review Laszlo and Peter. > LLAP cache is polluted by query-based compactor > --- > > Key: HIVE-22705 > URL: https://issues.apache.org/jira/browse/HIVE-22705 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22705.0.patch, HIVE-22705.1.patch, > HIVE-22705.2.patch > > > One of the steps that query-based compaction does is the verification of ACID > sort order by using the _validate_acid_sort_order_ UDF. This is a > prerequisite before the actual compaction can happen, and is done by a [query > that reads the whole table > content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167]. > This results in the whole table content being populated into the cache. The > problem is that this content is not useful and will rather pollute the cache > space, as it can never be used again: cache content binds to files (file IDs) > that obviously will be changed in this case by compaction. > I propose we disable LLAP caching in the session of query-based compaction's > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020874#comment-17020874 ] Eugene Chung commented on HIVE-22126: - I was wondering why this patch wasn't applied to 3.x. Is there any specific reason? > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
