[jira] [Commented] (HIVE-15222) replace org.json usage in ExplainTask/TezTask related classes with some alternative
[
https://issues.apache.org/jira/browse/HIVE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527218#comment-16527218
]
Hive QA commented on HIVE-15222:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12851538/HIVE-15222.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 05:54:54.284
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12236/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 05:54:54.288
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 5b2cbb5 HIVE-18786 : NPE in Hive windowing functions (Dongwook
Kwon via Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 5b2cbb5 HIVE-18786 : NPE in Hive windowing functions (Dongwook
Kwon via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 05:54:55.593
+ rm -rf ../yetus_PreCommit-HIVE-Build-12236
+ mkdir ../yetus_PreCommit-HIVE-Build-12236
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12236
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12236/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java:
does not exist in index
error:
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java: does
not exist in index
error: patch failed:
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java:27
Falling back to three-way merge...
Applied patch to
'common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java'
with conflicts.
error:
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java: does
not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java:20
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java'
with conflicts.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java:32
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java'
with conflicts.
error: src/java/org/apache/hadoop/hive/common/jsonexplain/JsonParser.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java: does
not exist in index
error:
src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParser.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java: does not exist
in index
error: src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java: does not exist in
index
error: java/org/apache/hadoop/hive/common/jsonexplain/JsonParser.java: does not
exist in index
error: java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java: does not
exist in index
error: java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java: does not
exist in index
error:
[jira] [Commented] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527216#comment-16527216
]
Hive QA commented on HIVE-18983:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919862/HIVE-18983.12.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 14630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_rename_table]
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_stats_status]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_3]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_alter_list_bucketing_table1]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like2]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_like_tbl_props]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_table_like_stats]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_table]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_ddl] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge3] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats12] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats13] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats14] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats15] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats2] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats6] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats7] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats8] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats9] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_noscan_1]
(batchId=60)
org.apache.hadoop.hive.cli.TestErasureCodingHDFSCliDriver.testCliDriver[erasure_simple]
(batchId=188)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries]
(batchId=101)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[whroot_external1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_1]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_noscan_1]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part8]
(batchId=137)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats12]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats13]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats14]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats15]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats2]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats6]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats7]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats8]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats9]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_noscan_1]
(batchId=133)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12235/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12235/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12235/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing
[jira] [Updated] (HIVE-19995) Aggregate row traffic for acid tables
[ https://issues.apache.org/jira/browse/HIVE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19995: Attachment: HIVE-19995.03.patch > Aggregate row traffic for acid tables > - > > Key: HIVE-19995 > URL: https://issues.apache.org/jira/browse/HIVE-19995 > Project: Hive > Issue Type: Sub-task > Components: Statistics, Transactions >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19995.01.patch, HIVE-19995.01wip01.patch, > HIVE-19995.01wip01.patch, HIVE-19995.01wip02.patch, HIVE-19995.02.patch, > HIVE-19995.02.patch, HIVE-19995.03.patch > > > for transactional tables we store basic stats in case of explicit > analyze/rewrite; but doesn't do anything in other caseswhich may even > lead to plans which oom... > It would be better to aggregate the total row traffic...because that is > already available; so that operator tree estimations could work with a real > upper bound of the row numbers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527196#comment-16527196
]
Aihua Xu commented on HIVE-18916:
-
[~stakiar] Right now we need to make tests clear before commit. Can you rebase
your code? Otherwise, the change looks good to me. +1.
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch, HIVE-18916.4.patch, HIVE-18916.5.patch, HIVE-18916.6.patch
>
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527173#comment-16527173
]
Hive QA commented on HIVE-18983:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
11s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 457 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
6s{color} | {color:red} The patch 499 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
17s{color} | {color:red} ql generated 1 new + 2287 unchanged - 0 fixed = 2288
total (was 2287) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Possible null pointer dereference of likeTable in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(ASTNode,
QB, SemanticAnalyzer$PlannerContext) Dereferenced at
SemanticAnalyzer.java:likeTable in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(ASTNode,
QB, SemanticAnalyzer$PlannerContext) Dereferenced at
SemanticAnalyzer.java:[line 13209] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12235/dev-support/hive-personality.sh
|
| git revision | master / 5b2cbb5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12235/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12235/yetus/whitespace-tabs.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12235/yetus/new-findbugs-ql.html
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12235/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12235/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add support for table properties inheritance in Create table like
>
[jira] [Commented] (HIVE-14925) MSCK repair table hang while running with multi threading enabled
[
https://issues.apache.org/jira/browse/HIVE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527138#comment-16527138
]
Hive QA commented on HIVE-14925:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12832969/HIVE-14925.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12234/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12234/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12234/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 03:47:57.242
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12234/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 03:47:57.246
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
1b3ac73..5b2cbb5 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 5b2cbb5 HIVE-18786 : NPE in Hive windowing functions (Dongwook
Kwon via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 03:47:59.050
+ rm -rf ../yetus_PreCommit-HIVE-Build-12234
+ mkdir ../yetus_PreCommit-HIVE-Build-12234
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12234
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12234/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java:426
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java' with
conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java:426
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12234
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12832969 - PreCommit-HIVE-Build
> MSCK repair table hang while running with multi threading enabled
> -
>
> Key: HIVE-14925
> URL: https://issues.apache.org/jira/browse/HIVE-14925
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.2.0
>Reporter: Ratheesh Kamoor
>Assignee: Ratheesh Kamoor
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: HIVE-14925.patch
>
>
> MSCK REPAIR TABLE hanging while running with multi-threading enabled
> (default). I think it is because of a major design flaw in how thread pool
> implemented in HiveMetaSoreChecker class / checkPartitionDirs method. This
> method has a thread pool which register Callable but callable makes a
> recursive call to checkPartitionDirs method again. This code will hang when
> number of directories is more than
[jira] [Commented] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527137#comment-16527137
]
Hive QA commented on HIVE-20013:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929456/HIVE-20013.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14628 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[kryo]
(batchId=165)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12233/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12233/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12233/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929456 - PreCommit-HIVE-Build
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18786) NPE in Hive windowing functions
[ https://issues.apache.org/jira/browse/HIVE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18786: Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Dongwook! > NPE in Hive windowing functions > --- > > Key: HIVE-18786 > URL: https://issues.apache.org/jira/browse/HIVE-18786 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2, 2.3.3 >Reporter: Michael Bieniosek >Assignee: Dongwook Kwon >Priority: Critical > Labels: patch > Fix For: 4.0.0 > > Attachments: HIVE-18786.1.patch, HIVE-18786.2.patch > > > When I run a Hive query with windowing functions, if there's enough data I > get an NPE. > For example something like this query might break: > select id, created_date, max(created_date) over (partition by id) > latest_created_any from ... > The only workaround I've found is to remove the windowing functions entirely. > The stacktrace looks suspiciously similar to +HIVE-15278+, but I'm in > hive-2.3.2 which appears to have the bugfix applied. > > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:297) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:365) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:287) > ... 16 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) > at > org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) > at > org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) > at > org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) > at > org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18786) NPE in Hive windowing functions
[ https://issues.apache.org/jira/browse/HIVE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527119#comment-16527119 ] Ashutosh Chauhan commented on HIVE-18786: - +1 > NPE in Hive windowing functions > --- > > Key: HIVE-18786 > URL: https://issues.apache.org/jira/browse/HIVE-18786 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2, 2.3.3 >Reporter: Michael Bieniosek >Assignee: Dongwook Kwon >Priority: Critical > Labels: patch > Attachments: HIVE-18786.1.patch, HIVE-18786.2.patch > > > When I run a Hive query with windowing functions, if there's enough data I > get an NPE. > For example something like this query might break: > select id, created_date, max(created_date) over (partition by id) > latest_created_any from ... > The only workaround I've found is to remove the windowing functions entirely. > The stacktrace looks suspiciously similar to +HIVE-15278+, but I'm in > hive-2.3.2 which appears to have the bugfix applied. > > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:297) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:365) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:287) > ... 16 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) > at > org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) > at > org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) > at > org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) > at > org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527114#comment-16527114
]
Hive QA commented on HIVE-20013:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12233/dev-support/hive-personality.sh
|
| git revision | master / 1b3ac73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12233/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
>
[jira] [Commented] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527100#comment-16527100 ] Prasanth Jayachandran commented on HIVE-20019: -- commons-logging and log4j jars are banned from being added as direct or transitive dependency. > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20019: - Attachment: HIVE-20019.1.patch > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20019: - Status: Patch Available (was: Open) > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14304) Beeline command will fail when entireLineAsCommand set to true
[
https://issues.apache.org/jira/browse/HIVE-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527086#comment-16527086
]
Hive QA commented on HIVE-14304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12819253/HIVE-14304.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12232/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12232/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12232/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 02:37:31.623
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12232/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 02:37:31.627
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 02:37:32.831
+ rm -rf ../yetus_PreCommit-HIVE-Build-12232
+ mkdir ../yetus_PreCommit-HIVE-Build-12232
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12232
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12232/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: beeline/src/java/org/apache/hive/beeline/Commands.java:1130
Falling back to three-way merge...
Applied patch to 'beeline/src/java/org/apache/hive/beeline/Commands.java' with
conflicts.
Going to apply patch with: git apply -p0
error: patch failed: beeline/src/java/org/apache/hive/beeline/Commands.java:1130
Falling back to three-way merge...
Applied patch to 'beeline/src/java/org/apache/hive/beeline/Commands.java' with
conflicts.
U beeline/src/java/org/apache/hive/beeline/Commands.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12232
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12819253 - PreCommit-HIVE-Build
> Beeline command will fail when entireLineAsCommand set to true
> --
>
> Key: HIVE-14304
> URL: https://issues.apache.org/jira/browse/HIVE-14304
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Niklaus Xiao
>Assignee: Hari Sankar Sivarama Subramaniyan
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-14304.1.patch
>
>
> Use beeline
> {code}
> beeline --entireLineAsCommand=true
> {code}
> show tables fail:
> {code}
> 0: jdbc:hive2://189.39.151.44:21066/> show tables;
> Error: Error while compiling statement: FAILED: ParseException line 1:11
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> {code}
> We should remove the trailing semi-colon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527085#comment-16527085
]
Hive QA commented on HIVE-15444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12843757/HIVE-15444.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14628 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12231/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12231/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12231/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12843757 - PreCommit-HIVE-Build
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch
>
>
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1481877998678_0011)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 100
> 0 0
> --
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
> --
> Loading data to table default.abc
> OK
> Time taken: 19.983 seconds
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set hive.execution.engine;
> hive.execution.engine=tez
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527045#comment-16527045
]
Misha Dmitriev edited comment on HIVE-19937 at 6/29/18 2:24 AM:
I took a quick look, and I am not sure this is done correctly. The code below
{code:java}
jobConf.forEach(entry -> {
StringInternUtils.internIfNotNull(entry.getKey());
StringInternUtils.internIfNotNull(entry.getValue());
}){code}
goes over each table entry and just invokes intern() for each key and value.
{{intern()}} returns an existing, "canonical" string for each string that is
duplicate. But the code doesn't store the returned strings back into the table.
To intern both keys and values in a hashtable, you typically need to create a
new table and effectively "intern and transfer" the contents from the old table
to the new table. Sometimes it may be possible to be more creative and actually
create a table with interned contents right away. Here it probably could be
done if you added some custom kryo deserialization code for such tables. But
maybe that's too big an effort.
As always, it would be good to measure how much memory was wasted before this
change and saved after it. This helps to prevent errors and to see how much was
actually achieved.
If {{jobConf}} is an instance of {{java.lang.Properties}}, and there are many
duplicates of such tables, then memory is wasted by both string contents of
these tables and by tables themselves (each table uses many extra Java objects
internally). So you may consider checking the
{{org.apache.hadoop.hive.common.CopyOnFirstWriteProperties}} class that I once
added for a somewhat similar use case.
was (Author: mi...@cloudera.com):
I took a quick look, and I am not sure this is done correctly. The code below
{code:java}
jobConf.forEach(entry -> {
StringInternUtils.internIfNotNull(entry.getKey());
StringInternUtils.internIfNotNull(entry.getValue());
}){code}
goes over each table entry and just invokes intern() for each key and value.
{{intern()}} returns an existing, "canonical" string for each string that is
duplicate. But the code doesn't store the returned strings back into the table.
To intern both keys and values in a hashtable, you typically need to create a
new table and effectively "intern and transfer" the contents from the old table
to the new table. Sometimes it may be possible to be more creative and actually
create a table with interned contents right away. Here it probably could be
done if you added some custom kryo deserialization code for such tables. But
maybe that's too big an effort.
As always, it would be good to see how much memory was wasted before this
change and saved after it. This helps to prevent errors and to see how much was
actually achieved.
If {{jobConf}} is an instance of {{java.lang.Properties}}, and there are many
duplicates of such tables, then memory is wasted by both string contents of
these tables and by tables themselves (each table uses many extra Java objects
internally). So you may consider checking the
{{org.apache.hadoop.hive.common.CopyOnFirstWriteProperties}} class that I once
added for a somewhat similar use case.
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527045#comment-16527045
]
Misha Dmitriev edited comment on HIVE-19937 at 6/29/18 2:23 AM:
I took a quick look, and I am not sure this is done correctly. The code below
{code:java}
jobConf.forEach(entry -> {
StringInternUtils.internIfNotNull(entry.getKey());
StringInternUtils.internIfNotNull(entry.getValue());
}){code}
goes over each table entry and just invokes intern() for each key and value.
{{intern()}} returns an existing, "canonical" string for each string that is
duplicate. But the code doesn't store the returned strings back into the table.
To intern both keys and values in a hashtable, you typically need to create a
new table and effectively "intern and transfer" the contents from the old table
to the new table. Sometimes it may be possible to be more creative and actually
create a table with interned contents right away. Here it probably could be
done if you added some custom kryo deserialization code for such tables. But
maybe that's too big an effort.
As always, it would be good to see how much memory was wasted before this
change and saved after it. This helps to prevent errors and to see how much was
actually achieved.
If {{jobConf}} is an instance of {{java.lang.Properties}}, and there are many
duplicates of such tables, then memory is wasted by both string contents of
these tables and by tables themselves (each table uses many extra Java objects
internally). So you may consider checking the
{{org.apache.hadoop.hive.common.CopyOnFirstWriteProperties}} class that I once
added for a somewhat similar use case.
was (Author: mi...@cloudera.com):
I took a quick look, and I am not sure this is done correctly. The code below
{code:java}
jobConf.forEach(entry -> {
StringInternUtils.internIfNotNull(entry.getKey());
StringInternUtils.internIfNotNull(entry.getValue());
}){code}
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527045#comment-16527045
]
Misha Dmitriev commented on HIVE-19937:
---
I took a quick look, and I am not sure this is done correctly. The code below
{code:java}
jobConf.forEach(entry -> {
StringInternUtils.internIfNotNull(entry.getKey());
StringInternUtils.internIfNotNull(entry.getValue());
}){code}
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15444) tez.queue.name is invalid after tez job running on CLI
[
https://issues.apache.org/jira/browse/HIVE-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527035#comment-16527035
]
Hive QA commented on HIVE-15444:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12231/dev-support/hive-personality.sh
|
| git revision | master / 1b3ac73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12231/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> tez.queue.name is invalid after tez job running on CLI
> --
>
> Key: HIVE-15444
> URL: https://issues.apache.org/jira/browse/HIVE-15444
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-15444.1.patch
>
>
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set tez.queue.name=HQ_OLPS;
> hive> set tez.queue.name;
> tez.queue.name=HQ_OLPS
> hive> insert into abc values(2,2);
> Query ID = hadoop_20161216181208_6c382e49-ac4a-4f52-ba1e-3ed962733fc1
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id
> application_1481877998678_0011)
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. container SUCCEEDED 1 100
> 0 0
> --
> VERTICES: 01/01 [==>>] 100% ELAPSED TIME: 6.57 s
> --
> Loading data to table default.abc
> OK
> Time taken: 19.983 seconds
> hive> set tez.queue.name;
> tez.queue.name is undefined
> hive> set hive.execution.engine;
> hive.execution.engine=tez
[jira] [Commented] (HIVE-19937) Intern JobConf objects in Spark tasks
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527015#comment-16527015
]
Sahil Takiar commented on HIVE-19937:
-
Thanks for the review Xuefu!
[~mi...@cloudera.com] could you take a quick look too, just want to make sure I
am doing this correctly. The high level explanation is that in Hive on Spark, a
single Spark executor (a single JVM) can run multiple Spark tasks in parallel.
Each task creates its own copy of {{JobConf}} (basically all the Hive
properties). This can cause significant overhead in the JVM, so this patch
basically interns all the entries in {{JobConf}}.
> Intern JobConf objects in Spark tasks
> -
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15223) replace org.json usage in EximUtil with some alternative
[
https://issues.apache.org/jira/browse/HIVE-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527016#comment-16527016
]
Hive QA commented on HIVE-15223:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12850960/HIVE-15223.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12230/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12230/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12230/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:27:37.076
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12230/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:27:37.079
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:27:37.663
+ rm -rf ../yetus_PreCommit-HIVE-Build-12230
+ mkdir ../yetus_PreCommit-HIVE-Build-12230
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12230
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12230/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:28
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java:28
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12230
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12850960 - PreCommit-HIVE-Build
> replace org.json usage in EximUtil with some alternative
>
>
> Key: HIVE-15223
> URL: https://issues.apache.org/jira/browse/HIVE-15223
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Teddy Choi
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-15223.1.patch
>
>
> The metadata is stored in json format...which changed lately with the advent
> of replication v2.
> I think jackson would be nice to have here - it could possibly aid to make
> this Metadata reading / writing more resilient against future serialization
> issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18811) Fix desc table, column comments are not displayed
[
https://issues.apache.org/jira/browse/HIVE-18811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527014#comment-16527014
]
Hive QA commented on HIVE-18811:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913564/changes.2
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12229/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12229/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12229/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:26:24.963
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12229/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:26:24.966
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:26:25.559
+ rm -rf ../yetus_PreCommit-HIVE-Build-12229
+ mkdir ../yetus_PreCommit-HIVE-Build-12229
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12229
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12229/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: unrecognized input
fatal: unrecognized input
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12229
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12913564 - PreCommit-HIVE-Build
> Fix desc table, column comments are not displayed
> -
>
> Key: HIVE-18811
> URL: https://issues.apache.org/jira/browse/HIVE-18811
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 1.2.1, 2.3.2
> Environment: CentOS 6.5
> Hive-1.2.1
> Hive-3.0.0
>Reporter: tartarus
>Assignee: tartarus
>Priority: Major
> Labels: patch
> Fix For: 3.2.0
>
> Attachments: HIVE_18811.2.patch, HIVE_18811.patch, changes, changes.2
>
>
> when column comment contain \t
> eg: CREATE TABLE `zhangmang_test`(`name` string COMMENT
> 'name{color:#ff}\t{color}zm');
> then execute : {color:#ff}desc zhangmang_test {color}
> {color:#ff}{color:#33}return :{color} name string
> name{color}
> because \t is the separator, so we should translate it
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20007) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-20007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527010#comment-16527010
]
Rui Li commented on HIVE-20007:
---
I see. Thanks [~jcamachorodriguez].
> Hive should carry out timestamp computations in UTC
> ---
>
> Key: HIVE-20007
> URL: https://issues.apache.org/jira/browse/HIVE-20007
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Ryan Blue
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Labels: timestamp
> Fix For: 4.0.0
>
> Attachments: HIVE-20007.patch
>
>
> Hive currently uses the "local" time of a java.sql.Timestamp to represent the
> SQL data type TIMESTAMP WITHOUT TIME ZONE. The purpose is to be able to use
> {{Timestamp#getYear()}} and similar methods to implement SQL functions like
> {{year}}.
> When the SQL session's time zone is a DST zone, such as America/Los_Angeles
> that alternates between PST and PDT, there are times that cannot be
> represented because the effective zone skips them.
> {code}
> hive> select TIMESTAMP '2015-03-08 02:10:00.101';
> 2015-03-08 03:10:00.101
> {code}
> Using UTC instead of the SQL session time zone as the underlying zone for a
> java.sql.Timestamp avoids this bug, while still returning correct values for
> {{getYear}} etc. Using UTC as the convenience representation (timestamp
> without time zone has no real zone) would make timestamp calculations more
> consistent and avoid similar problems in the future.
> Notably, this would break the {{unix_timestamp}} UDF that specifies the
> result is with respect to ["the default timezone and default
> locale"|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
> That function would need to be updated to use the
> {{System.getProperty("user.timezone")}} zone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18702) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
[
https://issues.apache.org/jira/browse/HIVE-18702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527012#comment-16527012
]
Hive QA commented on HIVE-18702:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12911562/HIVE-18702.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12228/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12228/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12228/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:25:16.429
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12228/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:25:16.433
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:25:17.012
+ rm -rf ../yetus_PreCommit-HIVE-Build-12228
+ mkdir ../yetus_PreCommit-HIVE-Build-12228
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12228
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12228/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: does not
exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1493
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1493
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12228
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12911562 - PreCommit-HIVE-Build
> INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting
> ---
>
> Key: HIVE-18702
> URL: https://issues.apache.org/jira/browse/HIVE-18702
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Major
> Fix For: 2.4.0, 3.2.0
>
> Attachments: HIVE-18702.1.patch, HIVE-18702.2.patch
>
>
> Enable Hive on TEZ. (MR works fine).
> *STEP 1. Create test data*
> {code}
> nano /home/test/users.txt
> {code}
> Add to file:
> {code}
> Peter,34
> John,25
> Mary,28
> {code}
> {code}
> hadoop fs -mkdir /bug
> hadoop fs -copyFromLocal /home/test/users.txt /bug
> hadoop fs -ls /bug
> {code}
> *EXPECTED RESULT:*
> {code}
> Found 2 items
>
> -rwxr-xr-x 3 root root 25 2015-10-15 16:11 /bug/users.txt
> {code}
> *STEP 2. Upload data to hive*
> {code}
> create external table bug(name string, age int) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY ','
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527009#comment-16527009
]
Sahil Takiar commented on HIVE-18916:
-
[~aihuaxu] any other comments on this?
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch, HIVE-18916.4.patch, HIVE-18916.5.patch, HIVE-18916.6.patch
>
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17033) Miss the jar when create slider package for llap
[
https://issues.apache.org/jira/browse/HIVE-17033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527008#comment-16527008
]
Hive QA commented on HIVE-17033:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12875699/HIVE-17033.001.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12227/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12227/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12227/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:24:04.760
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12227/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:24:04.763
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git clean -f -d
Removing itests/${project.basedir}/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-29 01:24:06.417
+ rm -rf ../yetus_PreCommit-HIVE-Build-12227
+ mkdir ../yetus_PreCommit-HIVE-Build-12227
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12227
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12227/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java:
does not exist in index
error: patch failed:
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java:397
Falling back to three-way merge...
Applied patch to
'llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java:397
Falling back to three-way merge...
Applied patch to
'llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java'
with conflicts.
U llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapServiceDriver.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12227
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12875699 - PreCommit-HIVE-Build
> Miss the jar when create slider package for llap
>
>
> Key: HIVE-17033
> URL: https://issues.apache.org/jira/browse/HIVE-17033
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 3.0.0
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17033.001.patch
>
>
> When create the slider package for llap, the jar for log4j-1.2-api is missed.
> The root cause is org.apache.log4j.NDC used to get jar of log4j-1.2-api, but
> this class is also existed in the jar of log4j. So, the jar of log4j-1.2-api
> won't be included.
> As a result, log4j-1.2-api-2.6.2.jar can't be found in llap-2.2.0-S.zip.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15944) The order of cols is error in ColumnPrunerReduceSinkProc because of sort operator
[
https://issues.apache.org/jira/browse/HIVE-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527007#comment-16527007
]
Hive QA commented on HIVE-15944:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856787/HIVE-15944.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14624 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=190)
[druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[with_column_pruner]
(batchId=9)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12226/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12226/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12226/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12856787 - PreCommit-HIVE-Build
> The order of cols is error in ColumnPrunerReduceSinkProc because of sort
> operator
> -
>
> Key: HIVE-15944
> URL: https://issues.apache.org/jira/browse/HIVE-15944
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.0, 2.2.0
>Reporter: wan kun
>Assignee: wan kun
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-15944-branch-1.1.patch, HIVE-15944.1.patch,
> HIVE-15944.2-branch-1.1.1.path.erroroutput, HIVE-15944.3.patch,
> HIVE-15944.4-branch-1.1.1.patch, HIVE-15944.4.patch, HIVE-15944.5.patch,
> HIVE-15944.6.patch, HIVE-15944.7.patch, HIVE-15944.8.patch, HIVE-15944.patch,
> STAGE_DEPENDENCIES
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> If one sql have two job.
> job 1:The order of cols is updated in ColumnPrunerReduceSinkProc because of
> sort operator.
> job 2 will read error in map operation because the cols order is old.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Attachment: HIVE-20032.1.patch > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-20032: --- Assignee: Sahil Takiar > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Status: Patch Available (was: Open) > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Description: Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, then we don't need to serialize the hashCode when shuffling data in HoS. > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Priority: Major > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Component/s: Spark > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Summary: Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled (was: Don) > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15944) The order of cols is error in ColumnPrunerReduceSinkProc because of sort operator
[
https://issues.apache.org/jira/browse/HIVE-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526972#comment-16526972
]
Hive QA commented on HIVE-15944:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12226/dev-support/hive-personality.sh
|
| git revision | master / 1b3ac73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12226/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> The order of cols is error in ColumnPrunerReduceSinkProc because of sort
> operator
> -
>
> Key: HIVE-15944
> URL: https://issues.apache.org/jira/browse/HIVE-15944
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.1.0, 2.2.0
>Reporter: wan kun
>Assignee: wan kun
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-15944-branch-1.1.patch, HIVE-15944.1.patch,
> HIVE-15944.2-branch-1.1.1.path.erroroutput, HIVE-15944.3.patch,
> HIVE-15944.4-branch-1.1.1.patch, HIVE-15944.4.patch, HIVE-15944.5.patch,
> HIVE-15944.6.patch, HIVE-15944.7.patch, HIVE-15944.8.patch, HIVE-15944.patch,
> STAGE_DEPENDENCIES
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> If one sql have two job.
> job 1:The order of cols is updated in ColumnPrunerReduceSinkProc because of
> sort operator.
> job 2 will read error in map operation because the cols order is old.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[
https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526950#comment-16526950
]
Hive QA commented on HIVE-19860:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929457/HIVE-19860.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14628 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.spark.client.rpc.TestRpc.testClientTimeout (batchId=314)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12225/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12225/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12225/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929457 - PreCommit-HIVE-Build
> HiveServer2 ObjectInspectorFactory memory leak with
> cachedUnionStructObjectInspector
>
>
> Key: HIVE-19860
> URL: https://issues.apache.org/jira/browse/HIVE-19860
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0
> Environment: hiveserver2 Interactive with LLAP.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot
> 2018-06-11 at 2.01.00 PM.png
>
>
> hiveserver2 is start seeing the memory pressure once the
> cachedUnionStructObjectInspector start going
> [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345]
> I did not see any eviction policy for cachedUnionStructObjectInspector, so we
> should implement some size or time-based eviction policy.
> !Screen Shot 2018-06-11 at 2.01.00 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19886) Logs may be directed to 2 files if --hiveconf hive.log.file is used
[
https://issues.apache.org/jira/browse/HIVE-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526934#comment-16526934
]
Jaume M commented on HIVE-19886:
[~kgyrtkirk] the choice of solutions are:
* Some initial logs would be sent to a different file until the reconfiguration
is done. These are the ones (since the Configuration object is created before
the reconfiguration happens):
{code}
2018-06-18T16:39:17,109 INFO [main] conf.MetastoreConf: Found configuration
file file:/hadoop-3.1.0/etc/hadoop/hive-site.xml
2018-06-18T16:39:17,364 INFO [main] conf.MetastoreConf: Unable to find config
file hivemetastore-site.xml
2018-06-18T16:39:17,364 INFO [main] conf.MetastoreConf: Found configuration
file null
2018-06-18T16:39:17,365 INFO [main] conf.MetastoreConf: Unable to find config
file metastore-site.xml
2018-06-18T16:39:17,365 INFO [main] conf.MetastoreConf: Found configuration
file null
{code}
* Warn in both files that a split is happening. Although I would do this as
well if we went with the first solution.
* Raise an exception
> Logs may be directed to 2 files if --hiveconf hive.log.file is used
> ---
>
> Key: HIVE-19886
> URL: https://issues.apache.org/jira/browse/HIVE-19886
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19886.2.patch, HIVE-19886.2.patch, HIVE-19886.patch
>
>
> hive launch script explicitly specific log4j2 configuration file to use. The
> main() methods in HiveServer2 and HiveMetastore reconfigures the logger based
> on user input via --hiveconf hive.log.file. This may cause logs to end up in
> 2 different files. Initial logs goes to the file specified in
> hive-log4j2.properties and after logger reconfiguration the rest of the logs
> goes to the file specified via --hiveconf hive.log.file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18929) The method humanReadableInt in HiveStringUtils.java has a race condition.
[
https://issues.apache.org/jira/browse/HIVE-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18929:
--
Attachment: HIVE-18929.1.patch
Status: Patch Available (was: Open)
> The method humanReadableInt in HiveStringUtils.java has a race condition.
> -
>
> Key: HIVE-18929
> URL: https://issues.apache.org/jira/browse/HIVE-18929
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 2.3.2
>Reporter: Chaiyong Ragkhitwetsagul
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18929.1.patch
>
>
> I found that the {{humanReadableInt(long number)}} method in the
> hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file
> contains code which has a race condition as shown in Hadoop (issue tracking
> ID HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix
> can also be seen in the Hadoop code base.
> I couldn't find a call to the method anywhere else in the code. But it might
> be worth to fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19409) Disable incremental rewriting with outdated materialized views
[ https://issues.apache.org/jira/browse/HIVE-19409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-19409. Resolution: Fixed Fix Version/s: 4.0.0 3.1.0 > Disable incremental rewriting with outdated materialized views > -- > > Key: HIVE-19409 > URL: https://issues.apache.org/jira/browse/HIVE-19409 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Critical > Fix For: 3.1.0, 4.0.0 > > > Add an option to disable incremental rewriting with outdated materialized > views. It will be disabled by default, and this will be an opt-in feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18118) Explain Extended should indicate if a file being read is an EC file
[ https://issues.apache.org/jira/browse/HIVE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526927#comment-16526927 ] Sahil Takiar commented on HIVE-18118: - +1 LGTM > Explain Extended should indicate if a file being read is an EC file > --- > > Key: HIVE-18118 > URL: https://issues.apache.org/jira/browse/HIVE-18118 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18118.1.patch, HIVE-18118.10.patch, > HIVE-18118.10.patch, HIVE-18118.11.patch, HIVE-18118.11.patch, > HIVE-18118.12.patch, HIVE-18118.2.patch, HIVE-18118.3.patch, > HIVE-18118.4.patch, HIVE-18118.5.patch, HIVE-18118.6.patch, > HIVE-18118.7.patch, HIVE-18118.8.patch, HIVE-18118.9.patch > > > We already print out the files Hive will read in the explain extended > command, we just have to modify it to say whether or not its an EC file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20005) acid_table_stats, acid_no_buckets, etc - query result change on the branch
[ https://issues.apache.org/jira/browse/HIVE-20005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526918#comment-16526918 ] Sergey Shelukhin commented on HIVE-20005: - [~steveyeom2017] can you look at this one? > acid_table_stats, acid_no_buckets, etc - query result change on the branch > -- > > Key: HIVE-20005 > URL: https://issues.apache.org/jira/browse/HIVE-20005 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Steve Yeom >Priority: Major > > Queries in the following tests have explain changes from running the query to > using stats (that is probably by design for most queries). > However, some of the queries also has the result changes; new results are > likely invalid. > TestCliDriver: acid_table_stats > TestMiniLlapLocalCliDriver: acid_no_buckets -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20029) add negative test for parallel insert
[ https://issues.apache.org/jira/browse/HIVE-20029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20029: Summary: add negative test for parallel insert (was: add negative tests) > add negative test for parallel insert > - > > Key: HIVE-20029 > URL: https://issues.apache.org/jira/browse/HIVE-20029 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > We need a few tests, esp. for parallel case, where we verify that stats are > NOT used. > Right now many code paths don't fail but return -1, null or whatever when > something else is not present, so positive tests might pass because they skip > the check, not because the check passes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19867) handle concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19867: Resolution: Fixed Fix Version/s: (was: 4.0.0) txnstats Status: Resolved (was: Patch Available) Committed patch to branch. It doesn't actually add an adequate test, I will file a followup for that > handle concurrent INSERTS > --- > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: txnstats > > Attachments: HIVE-19867.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19867) handle concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19867: Summary: handle concurrent INSERTS(was: Test and verify Concurrent INSERTS ) > handle concurrent INSERTS > --- > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19867.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[
https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526879#comment-16526879
]
Hive QA commented on HIVE-19860:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} serde in master has 193 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} serde: The patch generated 2 new + 12 unchanged - 1
fixed = 14 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12225/dev-support/hive-personality.sh
|
| git revision | master / 1b3ac73 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12225/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12225/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2 ObjectInspectorFactory memory leak with
> cachedUnionStructObjectInspector
>
>
> Key: HIVE-19860
> URL: https://issues.apache.org/jira/browse/HIVE-19860
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.1.0
> Environment: hiveserver2 Interactive with LLAP.
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot
> 2018-06-11 at 2.01.00 PM.png
>
>
> hiveserver2 is start seeing the memory pressure once the
> cachedUnionStructObjectInspector start going
> [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345]
> I did not see any eviction policy for cachedUnionStructObjectInspector, so we
> should implement some size or time-based eviction policy.
> !Screen Shot 2018-06-11 at 2.01.00 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18929) The method humanReadableInt in HiveStringUtils.java has a race condition.
[
https://issues.apache.org/jira/browse/HIVE-18929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18929:
-
Assignee: Andrew Sherman
> The method humanReadableInt in HiveStringUtils.java has a race condition.
> -
>
> Key: HIVE-18929
> URL: https://issues.apache.org/jira/browse/HIVE-18929
> Project: Hive
> Issue Type: Bug
> Components: API
>Affects Versions: 2.3.2
>Reporter: Chaiyong Ragkhitwetsagul
>Assignee: Andrew Sherman
>Priority: Minor
>
> I found that the {{humanReadableInt(long number)}} method in the
> hive/common/src/java/org/apache/hive/common/util/HiveStringUtils.java file
> contains code which has a race condition as shown in Hadoop (issue tracking
> ID HADOOP-9252: https://issues.apache.org/jira/browse/HADOOP-9252). The fix
> can also be seen in the Hadoop code base.
> I couldn't find a call to the method anywhere else in the code. But it might
> be worth to fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20027) TestRuntimeStats.testCleanup is flaky
[ https://issues.apache.org/jira/browse/HIVE-20027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526878#comment-16526878 ] Aihua Xu commented on HIVE-20027: - Resolve it now as dup. > TestRuntimeStats.testCleanup is flaky > - > > Key: HIVE-20027 > URL: https://issues.apache.org/jira/browse/HIVE-20027 > Project: Hive > Issue Type: Bug >Reporter: Aihua Xu >Priority: Major > > int deleted = objStore.deleteRuntimeStats(1); > assertEquals(1, deleted); > The testCleanup could fail if somehow there is GC pause before > deleteRuntimeStats happens so actually 2 stats will get deleted rather than > one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20027) TestRuntimeStats.testCleanup is flaky
[ https://issues.apache.org/jira/browse/HIVE-20027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu resolved HIVE-20027. - Resolution: Duplicate > TestRuntimeStats.testCleanup is flaky > - > > Key: HIVE-20027 > URL: https://issues.apache.org/jira/browse/HIVE-20027 > Project: Hive > Issue Type: Bug >Reporter: Aihua Xu >Priority: Major > > int deleted = objStore.deleteRuntimeStats(1); > assertEquals(1, deleted); > The testCleanup could fail if somehow there is GC pause before > deleteRuntimeStats happens so actually 2 stats will get deleted rather than > one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20031) Possible InflaterInputStream leaked in TezCommonUtils and related classes
[ https://issues.apache.org/jira/browse/HIVE-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M resolved HIVE-20031. Resolution: Invalid Release Note: Wrong board > Possible InflaterInputStream leaked in TezCommonUtils and related classes > - > > Key: HIVE-20031 > URL: https://issues.apache.org/jira/browse/HIVE-20031 > Project: Hive > Issue Type: Bug >Reporter: Jaume M >Priority: Major > > I don't think [this is > closed|https://github.com/apache/tez/blob/314dfc79b4b3f528b680b4fee73ad0dca3a3a19b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java#L397] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526867#comment-16526867
]
Hive QA commented on HIVE-18140:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929599/HIVE-18140.01-branch-3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14348 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=257)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=257)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=257)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=257)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=234)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=257)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=234)
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
(batchId=243)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=309)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12224/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12224/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12224/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929599 - PreCommit-HIVE-Build
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18140.01-branch-3.patch, HIVE-18140.01.patch,
> HIVE-18140.01wip01.patch, HIVE-18140.01wip03.patch, HIVE-18140.01wip04.patch,
> HIVE-18140.02.patch, HIVE-18140.02wip01.patch, HIVE-18140.03.patch,
> HIVE-18140.04.patch, HIVE-18140.04.patch, HIVE-18140.05.patch,
> HIVE-19140.02wip02.patch, HIVE-19727.02wip03.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20030) Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java and AnnotateRunTimeStatsOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-20030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-20030:
--
Attachment: HIVE-20030.1.patch
Status: Patch Available (was: Open)
> Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java
> and AnnotateRunTimeStatsOptimizer.java
>
>
> Key: HIVE-20030
> URL: https://issues.apache.org/jira/browse/HIVE-20030
> Project: Hive
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-20030.1.patch
>
>
> For some reason the Java compiler in IntellJ is more strict that the Oracle
> jdk compiler. Maybe this is something that can be configured away, but as it
> is simple I propose to make the code more type correct.
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
> Error:(613, 24) java: no suitable method found for
> findOperatorsUpstream(java.util.List extends
> org.apache.hadoop.hive.ql.plan.OperatorDesc>>,java.lang.Class)
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> org.apache.hadoop.hive.ql.exec.Operator))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(java.util.Collection>,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> java.util.Collection>))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class,java.util.Set)
> is not applicable
> (cannot infer type-variable(s) T
> (actual and formal argument lists differ in length))
> {code}
> and
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AnnotateRunTimeStatsOptimizer.java
> Error:(76, 12) java: no suitable method found for
> addAll(java.util.List>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(80, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(85, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> /Users/asherman/git/asf/hive2/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen/IntervalYearMonthScalarAddTimestampColumn.java
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-4367) enhance TRUNCATE syntax to drop data of external table
[ https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Hsu updated HIVE-4367: -- Summary: enhance TRUNCATE syntax to drop data of external table (was: enhance TRUNCATE syntax to drop data of external table) > enhance TRUNCATE syntax to drop data of external table > -- > > Key: HIVE-4367 > URL: https://issues.apache.org/jira/browse/HIVE-4367 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt, > HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch > > > In my use case , > sometimes I have to remove data of external tables to free up storage space > of the cluster . > So it's necessary for to enhance the syntax like > "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;" > to remove data from EXTERNAL table. > And I add a configuration property to enable remove data to Trash > > hive.truncate.skiptrash > false > > if true will remove data to trash, else false drop data immediately > > > For example : > hive (default)> TRUNCATE TABLE external1 partition (ds='11'); > FAILED: Error in semantic analysis: Cannot truncate non-managed table > external1 > hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE; > [2013-04-16 17:15:52]: Compile Start > [2013-04-16 17:15:52]: Compile End > [2013-04-16 17:15:52]: OK > [2013-04-16 17:15:52]: Time taken: 0.413 seconds > hive (default)> set hive.truncate.skiptrash; > hive.truncate.skiptrash=false > hive (default)> set hive.truncate.skiptrash=true; > hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE; > [2013-04-16 17:16:21]: Compile Start > [2013-04-16 17:16:21]: Compile End > [2013-04-16 17:16:21]: OK > [2013-04-16 17:16:21]: Time taken: 0.143 seconds > hive (default)> dfs -ls /user/test/.Trash/Current/; > Found 1 items > drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-4367) enhance TRUNCATE syntax to drop data of external table
[ https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anthony Hsu updated HIVE-4367: -- Summary: enhance TRUNCATE syntax to drop data of external table (was: enhance TRUNCATE syntex to drop data of external table) > enhance TRUNCATE syntax to drop data of external table > > > Key: HIVE-4367 > URL: https://issues.apache.org/jira/browse/HIVE-4367 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt, > HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch > > > In my use case , > sometimes I have to remove data of external tables to free up storage space > of the cluster . > So it's necessary for to enhance the syntax like > "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;" > to remove data from EXTERNAL table. > And I add a configuration property to enable remove data to Trash > > hive.truncate.skiptrash > false > > if true will remove data to trash, else false drop data immediately > > > For example : > hive (default)> TRUNCATE TABLE external1 partition (ds='11'); > FAILED: Error in semantic analysis: Cannot truncate non-managed table > external1 > hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE; > [2013-04-16 17:15:52]: Compile Start > [2013-04-16 17:15:52]: Compile End > [2013-04-16 17:15:52]: OK > [2013-04-16 17:15:52]: Time taken: 0.413 seconds > hive (default)> set hive.truncate.skiptrash; > hive.truncate.skiptrash=false > hive (default)> set hive.truncate.skiptrash=true; > hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE; > [2013-04-16 17:16:21]: Compile Start > [2013-04-16 17:16:21]: Compile End > [2013-04-16 17:16:21]: OK > [2013-04-16 17:16:21]: Time taken: 0.143 seconds > hive (default)> dfs -ls /user/test/.Trash/Current/; > Found 1 items > drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20030) Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java and AnnotateRunTimeStatsOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-20030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-20030:
-
> Fix Java compile errors that show up in IntelliJ from ConvertJoinMapJoin.java
> and AnnotateRunTimeStatsOptimizer.java
>
>
> Key: HIVE-20030
> URL: https://issues.apache.org/jira/browse/HIVE-20030
> Project: Hive
> Issue Type: Task
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> For some reason the Java compiler in IntellJ is more strict that the Oracle
> jdk compiler. Maybe this is something that can be configured away, but as it
> is simple I propose to make the code more type correct.
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java
> Error:(613, 24) java: no suitable method found for
> findOperatorsUpstream(java.util.List extends
> org.apache.hadoop.hive.ql.plan.OperatorDesc>>,java.lang.Class)
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> org.apache.hadoop.hive.ql.exec.Operator))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(java.util.Collection>,java.lang.Class)
> is not applicable
> (cannot infer type-variable(s) T
> (argument mismatch;
> java.util.List org.apache.hadoop.hive.ql.plan.OperatorDesc>> cannot be converted to
> java.util.Collection>))
> method
> org.apache.hadoop.hive.ql.exec.OperatorUtils.findOperatorsUpstream(org.apache.hadoop.hive.ql.exec.Operator,java.lang.Class,java.util.Set)
> is not applicable
> (cannot infer type-variable(s) T
> (actual and formal argument lists differ in length))
> {code}
> and
> {code}
> /Users/asherman/git/asf/hive2/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/AnnotateRunTimeStatsOptimizer.java
> Error:(76, 12) java: no suitable method found for
> addAll(java.util.List>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.List> cannot be
> converted to java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(80, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> Error:(85, 14) java: no suitable method found for
> addAll(java.util.Set>)
> method java.util.Collection.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> method java.util.Set.addAll(java.util.Collection org.apache.hadoop.hive.ql.exec.Operator org.apache.hadoop.hive.ql.plan.OperatorDesc>>) is not applicable
> (argument mismatch;
> java.util.Set> cannot be converted
> to java.util.Collection extends org.apache.hadoop.hive.ql.plan.OperatorDesc>>)
> /Users/asherman/git/asf/hive2/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen/IntervalYearMonthScalarAddTimestampColumn.java
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18140) Partitioned tables statistics can go wrong in basic stats mixed case
[
https://issues.apache.org/jira/browse/HIVE-18140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526794#comment-16526794
]
Hive QA commented on HIVE-18140:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-12224/patches/PreCommit-HIVE-Build-12224.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12224/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Partitioned tables statistics can go wrong in basic stats mixed case
>
>
> Key: HIVE-18140
> URL: https://issues.apache.org/jira/browse/HIVE-18140
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18140.01-branch-3.patch, HIVE-18140.01.patch,
> HIVE-18140.01wip01.patch, HIVE-18140.01wip03.patch, HIVE-18140.01wip04.patch,
> HIVE-18140.02.patch, HIVE-18140.02wip01.patch, HIVE-18140.03.patch,
> HIVE-18140.04.patch, HIVE-18140.04.patch, HIVE-18140.05.patch,
> HIVE-19140.02wip02.patch, HIVE-19727.02wip03.patch
>
>
> suppose the following scenario:
> * part1 has basic stats {{RC=10,DS=1K}}
> * all other partition has no basic stats (and a bunch of rows)
> then
> [this|https://github.com/apache/hive/blob/d9924ab3e285536f7e2cc15ecbea36a78c59c66d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L378]
> condition would be false; which in turn produces estimations for the whole
> partitioned table: {{RC=10,DS=1K}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20027) TestRuntimeStats.testCleanup is flaky
[ https://issues.apache.org/jira/browse/HIVE-20027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526792#comment-16526792 ] Peter Vary commented on HIVE-20027: --- [~aihuaxu]: Do you see it even after HIVE-19971? > TestRuntimeStats.testCleanup is flaky > - > > Key: HIVE-20027 > URL: https://issues.apache.org/jira/browse/HIVE-20027 > Project: Hive > Issue Type: Bug >Reporter: Aihua Xu >Priority: Major > > int deleted = objStore.deleteRuntimeStats(1); > assertEquals(1, deleted); > The testCleanup could fail if somehow there is GC pause before > deleteRuntimeStats happens so actually 2 stats will get deleted rather than > one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526791#comment-16526791 ] Sergey Shelukhin commented on HIVE-19975: - You should refresh the branch. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526786#comment-16526786
]
Hive QA commented on HIVE-17751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929435/HIVE-17751.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12223/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12223/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12223/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-28 21:17:21.428
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12223/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 21:17:21.432
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
777d62c..1b3ac73 master -> origin/master
417f016..26c5749e branch-3 -> origin/branch-3
+ git reset --hard HEAD
HEAD is now at 777d62c HIVE-19404: Revise DDL Task Result Logging (Andrew
Sherman, reviewed by Yongzhi Chen)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 1b3ac73 HIVE-20010: Fix create view over literals (Zoltan
Haindrich, reviewed by Ashutosh Chauhan, Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 21:17:23.494
+ rm -rf ../yetus_PreCommit-HIVE-Build-12223
+ mkdir ../yetus_PreCommit-HIVE-Build-12223
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12223
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12223/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 2560
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:1285
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java'
with conflicts.
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:77
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java'
with conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:6657: new blank line at EOF.
+
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:1285
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java'
with conflicts.
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:77
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java'
with conflicts.
U
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
U
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
warning: 1 line adds whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12223
+ exit 1
'
{noformat}
This message is
[jira] [Commented] (HIVE-18565) Set some timeout to all CliDriver tests
[
https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526784#comment-16526784
]
Hive QA commented on HIVE-18565:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929602/HIVE-18565.05.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 14432 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=261)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=261)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=190)
[druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=149)
[mapreduce2.q,orc_llap_counters1.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,parquet_struct_type_vectorization.q,orc_merge1.q,parquet_types_vectorization.q,orc_merge_diff_fs.q,llap_stats.q,llapdecider.q,llap_nullscan.q,orc_ppd_basic.q,rcfile_merge4.q,orc_merge3.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=151)
[intersect_all.q,unionDistinct_1.q,orc_ppd_schema_evol_3a.q,table_nonprintable.q,tez_union_dynamic_partition_2.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=152)
[import_exported_table.q,tez_union_dynamic_partition.q,except_distinct.q,orc_merge10.q,mapreduce1.q,mm_all.q,orc_merge4.q,dynamic_semijoin_user_level.q,temp_table_external.q,orc_struct_type_vectorization.q,llap_udf.q,schemeAuthority.q,external_table_with_space_in_location_path.q,external_table_purge.q,remote_script.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=153)
[parquet_complex_types_vectorization.q,cte_4.q,file_with_header_footer.q,explainuser_2.q,rcfile_merge2.q,bucket5.q,parquet_map_type_vectorization.q,insert_into2.q,reduce_deduplicate.q,orc_llap_counters.q,schemeAuthority2.q,load_fs2.q,reduce_deduplicate_distinct.q,rcfile_merge3.q,intersect_distinct.q]
TestMiniTezCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=105)
[update_orig_table.q,explainanalyze_3.q,orc_vectorization_ppd.q,vector_non_string_partition.q,multi_count_distinct.q,explainanalyze_1.q,explainuser_3.q,explainanalyze_4.q,hybridgrace_hashjoin_1.q,delete_orig_table.q,hybridgrace_hashjoin_2.q,tez-tag.q,orc_merge12.q,explainanalyze_5.q,vector_join_part_col_char.q]
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=261)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=261)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_with_masking]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=184)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/1/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929602 - PreCommit-HIVE-Build
> Set some timeout to all CliDriver tests
> ---
>
> Key: HIVE-18565
> URL: https://issues.apache.org/jira/browse/HIVE-18565
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18565.01.patch, HIVE-18565.02.patch,
> HIVE-18565.03.patch, HIVE-18565.04.patch, HIVE-18565.05.patch,
> HIVE-18565.05.patch
>
>
> if a testcase runs into an infinite loop or something ; it should still fail
> instead of causing timeout at the ptest executor
--
This message was sent by
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[
https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526776#comment-16526776
]
Steve Yeom commented on HIVE-19975:
---
[~sershe] I checked with "stats_part.q". Yes it is calling
HiveMetaStore::get_partitions_statistics_req method.
But your additional change to AcidUtils.getTableSnapshot():
to disable to get "validWriteIdList" by HIVE_IN_TEST check:
if (validWriteIdList == null && !HiveConf.getBoolVar(conf,
ConfVars.HIVE_IN_TEST)) {
it passes null.
But according to [~jcamachorodriguez] i think you understand it is ok when we
call this:
validWriteIdList = getTableValidWriteIdListWithTxnList(..)
I think you jumped into conclusion that multiple inserts for partitioned table
case is not working at
stats_part.q due to validWriteIdList == null. But that is not the case.
> Checking writeIdList per table may not check the commit level of a partition
> on a partitioned table
> ---
>
> Key: HIVE-19975
> URL: https://issues.apache.org/jira/browse/HIVE-19975
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
>
> writeIdList is per table entity but stats for a partitioned table are per
> partition.
> I.e., each record in PARTITIONS has an independent stats.
> So if we check the validity of a partition's stats, we need to check in the
> context of
> a partiton.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19404) Revise DDL Task Result Logging
[
https://issues.apache.org/jira/browse/HIVE-19404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526767#comment-16526767
]
Andrew Sherman commented on HIVE-19404:
---
Thanks [~ychena]
> Revise DDL Task Result Logging
> --
>
> Key: HIVE-19404
> URL: https://issues.apache.org/jira/browse/HIVE-19404
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Trivial
> Labels: noob
> Fix For: 4.0.0
>
> Attachments: HIVE-19404.1.patch
>
>
> There is some logging in {{DDLTask}} that can be made better:
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: results : 706
> {code}
> This logging should either be demoted to _debug_ level logging and/or
> requires additional context.
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: Found 706 tables that match the
> SHOW DATABASE statement
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19867) Test and verify Concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-19867: -- Status: Patch Available (was: Open) > Test and verify Concurrent INSERTS > > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19867.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19867) Test and verify Concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526755#comment-16526755 ] Steve Yeom commented on HIVE-19867: --- patch 01 is based on commit 61c55a3f66a71b7ff513fc45d68b749c061fa820 Merge: 8f32832428 b5160e7441 Author: sergey Date: Mon Jun 25 21:02:25 2018 -0700 HIVE-19416 : merge master into branch (Sergey Shelukhin) It may fail due to conflicts > Test and verify Concurrent INSERTS > > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19867.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19867) Test and verify Concurrent INSERTS
[ https://issues.apache.org/jira/browse/HIVE-19867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-19867: -- Attachment: HIVE-19867.01.patch > Test and verify Concurrent INSERTS > > > Key: HIVE-19867 > URL: https://issues.apache.org/jira/browse/HIVE-19867 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19867.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20010) Fix create view over literals
[
https://issues.apache.org/jira/browse/HIVE-20010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-20010:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 4.0.0
3.1.0
Status: Resolved (was: Patch Available)
Patch pushed to master/branch-3. Thanks [~kgyrtkirk]!
> Fix create view over literals
> -
>
> Key: HIVE-20010
> URL: https://issues.apache.org/jira/browse/HIVE-20010
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-20010.01.patch
>
>
> {code}
> create or replace view v1 as select 1 as q
> {code}
> results in an error:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table already exists:
> default.v1 (state=08S01,code=1)
> {code}
> however the following works (thank you [~mgergely])
> {code}
> create or replace view v1 as select 1 as q union all select 1 as qq where
> false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18565) Set some timeout to all CliDriver tests
[
https://issues.apache.org/jira/browse/HIVE-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526742#comment-16526742
]
Hive QA commented on HIVE-18565:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-1/dev-support/hive-personality.sh
|
| git revision | master / 777d62c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests/util itests/qtest-spark U: itests |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-1/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Set some timeout to all CliDriver tests
> ---
>
> Key: HIVE-18565
> URL: https://issues.apache.org/jira/browse/HIVE-18565
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18565.01.patch, HIVE-18565.02.patch,
> HIVE-18565.03.patch, HIVE-18565.04.patch, HIVE-18565.05.patch,
> HIVE-18565.05.patch
>
>
> if a testcase runs into an infinite loop or something ; it should still fail
> instead of causing timeout at the ptest executor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19995) Aggregate row traffic for acid tables
[
https://issues.apache.org/jira/browse/HIVE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526722#comment-16526722
]
Hive QA commented on HIVE-19995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929600/HIVE-19995.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12221/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12221/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12221/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929600/HIVE-19995.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929600 - PreCommit-HIVE-Build
> Aggregate row traffic for acid tables
> -
>
> Key: HIVE-19995
> URL: https://issues.apache.org/jira/browse/HIVE-19995
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics, Transactions
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19995.01.patch, HIVE-19995.01wip01.patch,
> HIVE-19995.01wip01.patch, HIVE-19995.01wip02.patch, HIVE-19995.02.patch,
> HIVE-19995.02.patch
>
>
> for transactional tables we store basic stats in case of explicit
> analyze/rewrite; but doesn't do anything in other caseswhich may even
> lead to plans which oom...
> It would be better to aggregate the total row traffic...because that is
> already available; so that operator tree estimations could work with a real
> upper bound of the row numbers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526720#comment-16526720
]
Hive QA commented on HIVE-16950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929576/HIVE-16950.10.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12220/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12220/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12220/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929576/HIVE-16950.10.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929576 - PreCommit-HIVE-Build
> Dropping hive database/table which was created explicitly in default database
> location, deletes all databases data from default database location
> -
>
> Key: HIVE-16950
> URL: https://issues.apache.org/jira/browse/HIVE-16950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Rahul Kalgunde
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HIVE-16950.1.patch, HIVE-16950.10.patch,
> HIVE-16950.2.patch, HIVE-16950.3.patch, HIVE-16950.4.patch,
> HIVE-16950.5.patch, HIVE-16950.6.patch, HIVE-16950.7.patch,
> HIVE-16950.8.patch, HIVE-16950.9.patch
>
>
> When database/table is created explicitly pointing to the default location,
> dropping the database/table deletes all the data associated with the all
> databases/tables.
> Steps to replicate:
> in below e.g. dropping table test_db2 also deletes data of test_db1 where as
> metastore still contains test_db1
> hive> create database test_db1;
> OK
> Time taken: 4.858 seconds
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.599 seconds, Fetched: 1 row(s)
> hive> create database test_db2 location '/apps/hive/warehouse' ;
> OK
> Time taken: 1.457 seconds
> hive> describe database test_db2;
> OK
> test_db2
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse rootUSER
> Time taken: 0.582 seconds, Fetched: 1 row(s)
> hive> drop database test_db2;
> OK
> Time taken: 1.317 seconds
> hive> dfs -ls /apps/hive/warehouse;
> ls: `/apps/hive/warehouse': No such file or directory
> Command failed with exit code = 1
> Query returned non-zero code: 1, cause: null
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.629 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526717#comment-16526717
]
Hive QA commented on HIVE-16950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929576/HIVE-16950.10.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12219/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12219/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929576/HIVE-16950.10.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929576 - PreCommit-HIVE-Build
> Dropping hive database/table which was created explicitly in default database
> location, deletes all databases data from default database location
> -
>
> Key: HIVE-16950
> URL: https://issues.apache.org/jira/browse/HIVE-16950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Rahul Kalgunde
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HIVE-16950.1.patch, HIVE-16950.10.patch,
> HIVE-16950.2.patch, HIVE-16950.3.patch, HIVE-16950.4.patch,
> HIVE-16950.5.patch, HIVE-16950.6.patch, HIVE-16950.7.patch,
> HIVE-16950.8.patch, HIVE-16950.9.patch
>
>
> When database/table is created explicitly pointing to the default location,
> dropping the database/table deletes all the data associated with the all
> databases/tables.
> Steps to replicate:
> in below e.g. dropping table test_db2 also deletes data of test_db1 where as
> metastore still contains test_db1
> hive> create database test_db1;
> OK
> Time taken: 4.858 seconds
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.599 seconds, Fetched: 1 row(s)
> hive> create database test_db2 location '/apps/hive/warehouse' ;
> OK
> Time taken: 1.457 seconds
> hive> describe database test_db2;
> OK
> test_db2
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse rootUSER
> Time taken: 0.582 seconds, Fetched: 1 row(s)
> hive> drop database test_db2;
> OK
> Time taken: 1.317 seconds
> hive> dfs -ls /apps/hive/warehouse;
> ls: `/apps/hive/warehouse': No such file or directory
> Command failed with exit code = 1
> Query returned non-zero code: 1, cause: null
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.629 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19995) Aggregate row traffic for acid tables
[
https://issues.apache.org/jira/browse/HIVE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526715#comment-16526715
]
Hive QA commented on HIVE-19995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929600/HIVE-19995.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12218/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12218/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12218/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-28 19:51:54.459
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12218/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 19:51:54.462
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
ae008b7..777d62c master -> origin/master
+ git reset --hard HEAD
HEAD is now at ae008b7 HIVE-20007: Hive should carry out timestamp computations
in UTC (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 777d62c HIVE-19404: Revise DDL Task Result Logging (Andrew
Sherman, reviewed by Yongzhi Chen)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 19:51:56.293
+ rm -rf ../yetus_PreCommit-HIVE-Build-12218
+ mkdir ../yetus_PreCommit-HIVE-Build-12218
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12218
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12218/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/test/results/clientpositive/acid_table_stats.q.out:133
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/acid_table_stats.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out:665
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out:95
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/insert_values_orig_table_use_metadata.q.out:170
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/insert_values_orig_table_use_metadata.q.out'
with conflicts.
error: patch failed: ql/src/test/results/clientpositive/row__id.q.out:62
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/row__id.q.out' with
conflicts.
error: patch failed:
ql/src/test/results/clientpositive/tez/acid_vectorization_original_tez.q.out:680
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/tez/acid_vectorization_original_tez.q.out'
with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:64: trailing whitespace.
explain merge into t as t using upd_t as u ON t.a = u.a
/data/hiveptest/working/scratch/build.patch:68: trailing whitespace.
merge into t as t using upd_t as u ON t.a = u.a
/data/hiveptest/working/scratch/build.patch:77: trailing whitespace.
merge into t as t using upd_t as u ON t.a = u.a
[jira] [Commented] (HIVE-20010) Fix create view over literals
[
https://issues.apache.org/jira/browse/HIVE-20010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526713#comment-16526713
]
Hive QA commented on HIVE-20010:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929408/HIVE-20010.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14628 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12217/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12217/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929408 - PreCommit-HIVE-Build
> Fix create view over literals
> -
>
> Key: HIVE-20010
> URL: https://issues.apache.org/jira/browse/HIVE-20010
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20010.01.patch
>
>
> {code}
> create or replace view v1 as select 1 as q
> {code}
> results in an error:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table already exists:
> default.v1 (state=08S01,code=1)
> {code}
> however the following works (thank you [~mgergely])
> {code}
> create or replace view v1 as select 1 as q union all select 1 as qq where
> false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18786) NPE in Hive windowing functions
[ https://issues.apache.org/jira/browse/HIVE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongwook Kwon updated HIVE-18786: - Attachment: HIVE-18786.2.patch > NPE in Hive windowing functions > --- > > Key: HIVE-18786 > URL: https://issues.apache.org/jira/browse/HIVE-18786 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2, 2.3.3 >Reporter: Michael Bieniosek >Assignee: Dongwook Kwon >Priority: Critical > Labels: patch > Attachments: HIVE-18786.1.patch, HIVE-18786.2.patch > > > When I run a Hive query with windowing functions, if there's enough data I > get an NPE. > For example something like this query might break: > select id, created_date, max(created_date) over (partition by id) > latest_created_any from ... > The only workaround I've found is to remove the windowing functions entirely. > The stacktrace looks suspiciously similar to +HIVE-15278+, but I'm in > hive-2.3.2 which appears to have the bugfix applied. > > Caused by: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:297) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317) > at > org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) > ... 14 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row (tag=0) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:365) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:287) > ... 16 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) > at > org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) > at > org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) > at > org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) > at > org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) > at > org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) > at > org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19820) add ACID stats support to background stats updater
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526699#comment-16526699 ] Sergey Shelukhin commented on HIVE-19820: - [~hagleitn] Eugene is out, can you review the rest? > add ACID stats support to background stats updater > -- > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.02-master-txnstats.patch, HIVE-19820.03-master-txnstats.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20016) Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20016: Summary: Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure (was: Investigate random test failure ) > Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure > --- > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20016.1.patch > > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19404) Revise DDL Task Result Logging
[
https://issues.apache.org/jira/browse/HIVE-19404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-19404:
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~asherman] for your contribution.
> Revise DDL Task Result Logging
> --
>
> Key: HIVE-19404
> URL: https://issues.apache.org/jira/browse/HIVE-19404
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Trivial
> Labels: noob
> Fix For: 4.0.0
>
> Attachments: HIVE-19404.1.patch
>
>
> There is some logging in {{DDLTask}} that can be made better:
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: results : 706
> {code}
> This logging should either be demoted to _debug_ level logging and/or
> requires additional context.
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: Found 706 tables that match the
> SHOW DATABASE statement
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20028) Metastore client cache config is used incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526689#comment-16526689 ] Sergey Shelukhin commented on HIVE-20028: - +1. Should go into branch-3 > Metastore client cache config is used incorrectly > - > > Key: HIVE-20028 > URL: https://issues.apache.org/jira/browse/HIVE-20028 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20028.1.patch > > > Metastore client cache config is not used correctly. Enabling the cache > actually disables it and vice versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20010) Fix create view over literals
[
https://issues.apache.org/jira/browse/HIVE-20010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526690#comment-16526690
]
Hive QA commented on HIVE-20010:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12217/dev-support/hive-personality.sh
|
| git revision | master / ae008b7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12217/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix create view over literals
> -
>
> Key: HIVE-20010
> URL: https://issues.apache.org/jira/browse/HIVE-20010
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20010.01.patch
>
>
> {code}
> create or replace view v1 as select 1 as q
> {code}
> results in an error:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table already exists:
> default.v1 (state=08S01,code=1)
> {code}
> however the following works (thank you [~mgergely])
> {code}
> create or replace view v1 as select 1 as q union all select 1 as qq where
> false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20028) Metastore client cache config is used incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526684#comment-16526684 ] Prasanth Jayachandran commented on HIVE-20028: -- The change in HiveStreamingConnection is, it is supposed to set whatever boolean received as argument but it was always setting to true resulting in metastore client cache not disabled properly. [~sershe] can you please take a look? 2 line change. > Metastore client cache config is used incorrectly > - > > Key: HIVE-20028 > URL: https://issues.apache.org/jira/browse/HIVE-20028 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20028.1.patch > > > Metastore client cache config is not used correctly. Enabling the cache > actually disables it and vice versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526682#comment-16526682 ] Sergey Shelukhin commented on HIVE-20021: - +1 pending tests > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20028) Metastore client cache config is used incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20028: - Status: Patch Available (was: Open) > Metastore client cache config is used incorrectly > - > > Key: HIVE-20028 > URL: https://issues.apache.org/jira/browse/HIVE-20028 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20028.1.patch > > > Metastore client cache config is not used correctly. Enabling the cache > actually disables it and vice versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20028) Metastore client cache config is used incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20028: - Attachment: HIVE-20028.1.patch > Metastore client cache config is used incorrectly > - > > Key: HIVE-20028 > URL: https://issues.apache.org/jira/browse/HIVE-20028 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20028.1.patch > > > Metastore client cache config is not used correctly. Enabling the cache > actually disables it and vice versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20028) Metastore client cache config is used incorrectly
[ https://issues.apache.org/jira/browse/HIVE-20028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20028: > Metastore client cache config is used incorrectly > - > > Key: HIVE-20028 > URL: https://issues.apache.org/jira/browse/HIVE-20028 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Metastore client cache config is not used correctly. Enabling the cache > actually disables it and vice versa. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20016) Investigate random test failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20016: Attachment: HIVE-20016.1.patch > Investigate random test failure > > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20016.1.patch > > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20016) Investigate random test failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20016: Status: Patch Available (was: Open) > Investigate random test failure > > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20016.1.patch > > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19948) HiveCli is not splitting the command by semicolon properly if quotes are inside the string
[
https://issues.apache.org/jira/browse/HIVE-19948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526672#comment-16526672
]
Aihua Xu commented on HIVE-19948:
-
[~hagleitn] I may have misunderstood. So should we retry to get the flaky test
to pass or should we get the flaky test to get fixed completely?
> HiveCli is not splitting the command by semicolon properly if quotes are
> inside the string
> ---
>
> Key: HIVE-19948
> URL: https://issues.apache.org/jira/browse/HIVE-19948
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19948.1.patch, HIVE-19948.2.patch,
> HIVE-19948.3.patch
>
>
> HIVE-15297 tries to split the command by considering semicolon inside string,
> but it doesn't consider the case that quotes can also be inside string.
> For the following command {{insert into escape1 partition (ds='1', part='3')
> values ("abc' ");}}, it will fail with
> {noformat}
> 18/06/19 16:37:05 ERROR ql.Driver: FAILED: ParseException line 1:64
> extraneous input ';' expecting EOF near ''
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:64 extraneous input
> ';' expecting EOF near ''
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:220)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:606)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1686)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1633)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1628)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526670#comment-16526670 ] Sergey Shelukhin commented on HIVE-19418: - Sure, works for me. I feel these calls (with nulls, etc) are silly enough that noone should be intentionally handling exceptions for them, so it can throw anything. Please don't revert the functionality of this patch though... > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.1.0, 4.0.0 > > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.03.patch, HIVE-19418.04.patch, HIVE-19418.05.patch, > HIVE-19418.06.patch, HIVE-19418.06.patch, HIVE-19418.07.patch, > HIVE-19418.07.patch, HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526668#comment-16526668 ] Sergey Shelukhin commented on HIVE-19975: - It's used in execution part for StatsOptimizer. It receives request with table snapshot set, but doesn't pass it on. See the latest commit on the branch that addresses that. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20011) Move away from append mode in proto logging hook
[
https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526659#comment-16526659
]
Hive QA commented on HIVE-20011:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929400/HIVE-20011.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14628 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12216/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12216/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12216/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929400 - PreCommit-HIVE-Build
> Move away from append mode in proto logging hook
>
>
> Key: HIVE-20011
> URL: https://issues.apache.org/jira/browse/HIVE-20011
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-20011.01.patch
>
>
> Proto logging hook uses append mode with SequenceFile which does not work on
> filesystems which does not support append. Move away to using sequence file
> with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond
> the length in NameNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-4367) enhance TRUNCATE syntex to drop data of external table
[ https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526656#comment-16526656 ] Jason Dere commented on HIVE-4367: -- Attaching same patch again > enhance TRUNCATE syntex to drop data of external table > > > Key: HIVE-4367 > URL: https://issues.apache.org/jira/browse/HIVE-4367 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt, > HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch > > > In my use case , > sometimes I have to remove data of external tables to free up storage space > of the cluster . > So it's necessary for to enhance the syntax like > "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;" > to remove data from EXTERNAL table. > And I add a configuration property to enable remove data to Trash > > hive.truncate.skiptrash > false > > if true will remove data to trash, else false drop data immediately > > > For example : > hive (default)> TRUNCATE TABLE external1 partition (ds='11'); > FAILED: Error in semantic analysis: Cannot truncate non-managed table > external1 > hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE; > [2013-04-16 17:15:52]: Compile Start > [2013-04-16 17:15:52]: Compile End > [2013-04-16 17:15:52]: OK > [2013-04-16 17:15:52]: Time taken: 0.413 seconds > hive (default)> set hive.truncate.skiptrash; > hive.truncate.skiptrash=false > hive (default)> set hive.truncate.skiptrash=true; > hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE; > [2013-04-16 17:16:21]: Compile Start > [2013-04-16 17:16:21]: Compile End > [2013-04-16 17:16:21]: OK > [2013-04-16 17:16:21]: Time taken: 0.143 seconds > hive (default)> dfs -ls /user/test/.Trash/Current/; > Found 1 items > drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-4367) enhance TRUNCATE syntex to drop data of external table
[ https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-4367: - Attachment: HIVE-4367.6.patch > enhance TRUNCATE syntex to drop data of external table > > > Key: HIVE-4367 > URL: https://issues.apache.org/jira/browse/HIVE-4367 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: caofangkun >Assignee: caofangkun >Priority: Minor > Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt, > HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch > > > In my use case , > sometimes I have to remove data of external tables to free up storage space > of the cluster . > So it's necessary for to enhance the syntax like > "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;" > to remove data from EXTERNAL table. > And I add a configuration property to enable remove data to Trash > > hive.truncate.skiptrash > false > > if true will remove data to trash, else false drop data immediately > > > For example : > hive (default)> TRUNCATE TABLE external1 partition (ds='11'); > FAILED: Error in semantic analysis: Cannot truncate non-managed table > external1 > hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE; > [2013-04-16 17:15:52]: Compile Start > [2013-04-16 17:15:52]: Compile End > [2013-04-16 17:15:52]: OK > [2013-04-16 17:15:52]: Time taken: 0.413 seconds > hive (default)> set hive.truncate.skiptrash; > hive.truncate.skiptrash=false > hive (default)> set hive.truncate.skiptrash=true; > hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE; > [2013-04-16 17:16:21]: Compile Start > [2013-04-16 17:16:21]: Compile End > [2013-04-16 17:16:21]: OK > [2013-04-16 17:16:21]: Time taken: 0.143 seconds > hive (default)> dfs -ls /user/test/.Trash/Current/; > Found 1 items > drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19967) SMB Join : Need Optraits for PTFOperator ala GBY Op
[ https://issues.apache.org/jira/browse/HIVE-19967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19967: -- Attachment: HIVE-19967.8.patch > SMB Join : Need Optraits for PTFOperator ala GBY Op > --- > > Key: HIVE-19967 > URL: https://issues.apache.org/jira/browse/HIVE-19967 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19967.1.patch, HIVE-19967.2.patch, > HIVE-19967.3.patch, HIVE-19967.4.patch, HIVE-19967.5.patch, > HIVE-19967.6.patch, HIVE-19967.7.patch, HIVE-19967.8.patch > > > The SMB join on one or more PTF Ops should reset the optraits keys just like > GBY Op does. > Currently there is no implementation of PTFOp optraits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19948) HiveCli is not splitting the command by semicolon properly if quotes are inside the string
[
https://issues.apache.org/jira/browse/HIVE-19948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526641#comment-16526641
]
Gunther Hagleitner commented on HIVE-19948:
---
[~aihuaxu] there was a vote recently saying we won't commit w/o a green run.
Why does this not apply here?
> HiveCli is not splitting the command by semicolon properly if quotes are
> inside the string
> ---
>
> Key: HIVE-19948
> URL: https://issues.apache.org/jira/browse/HIVE-19948
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19948.1.patch, HIVE-19948.2.patch,
> HIVE-19948.3.patch
>
>
> HIVE-15297 tries to split the command by considering semicolon inside string,
> but it doesn't consider the case that quotes can also be inside string.
> For the following command {{insert into escape1 partition (ds='1', part='3')
> values ("abc' ");}}, it will fail with
> {noformat}
> 18/06/19 16:37:05 ERROR ql.Driver: FAILED: ParseException line 1:64
> extraneous input ';' expecting EOF near ''
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:64 extraneous input
> ';' expecting EOF near ''
> at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:220)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:74)
> at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:67)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:606)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1686)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1633)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1628)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19938) Upgrade scripts for information schema
[ https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526639#comment-16526639 ] Daniel Dai commented on HIVE-19938: --- Fix ptest failure. > Upgrade scripts for information schema > -- > > Key: HIVE-19938 > URL: https://issues.apache.org/jira/browse/HIVE-19938 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch, > HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch, > HIVE-19938.6.patch, HIVE-19938.7.patch > > > To make schematool -upgradeSchema work for information schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19938) Upgrade scripts for information schema
[ https://issues.apache.org/jira/browse/HIVE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-19938: -- Attachment: HIVE-19938.7.patch > Upgrade scripts for information schema > -- > > Key: HIVE-19938 > URL: https://issues.apache.org/jira/browse/HIVE-19938 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19938.1.patch, HIVE-19938.2.patch, > HIVE-19938.3.patch, HIVE-19938.4.patch, HIVE-19938.5.patch, > HIVE-19938.6.patch, HIVE-19938.7.patch > > > To make schematool -upgradeSchema work for information schema. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19981) Managed tables converted to external tables by the HiveStrictManagedMigration utility should be set to delete data when the table is dropped
[ https://issues.apache.org/jira/browse/HIVE-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-19981: -- Resolution: Fixed Fix Version/s: 4.0.0 Target Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to master > Managed tables converted to external tables by the HiveStrictManagedMigration > utility should be set to delete data when the table is dropped > > > Key: HIVE-19981 > URL: https://issues.apache.org/jira/browse/HIVE-19981 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19981.1.patch > > > Using the HiveStrictManagedMigration utility, tables can be converted to > conform to the Hive strict managed tables mode. > For managed tables that are converted to external tables by the utility, > these tables should keep the "drop data on delete" semantics they had when > they were managed tables. > One way to do this is to introduce a table property "external.table.purge", > which if true (and if the table is an external table), will let Hive know to > delete the table data when the table is dropped. This property will be set by > the HiveStrictManagedMigration utility when managed tables are converted to > external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20011) Move away from append mode in proto logging hook
[
https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526617#comment-16526617
]
Hive QA commented on HIVE-20011:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2287 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12216/dev-support/hive-personality.sh
|
| git revision | master / ae008b7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12216/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move away from append mode in proto logging hook
>
>
> Key: HIVE-20011
> URL: https://issues.apache.org/jira/browse/HIVE-20011
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-20011.01.patch
>
>
> Proto logging hook uses append mode with SequenceFile which does not work on
> filesystems which does not support append. Move away to using sequence file
> with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond
> the length in NameNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526615#comment-16526615 ] Steve Yeom commented on HIVE-19975: --- [~sershe] I think a quick verification may be that with your check please run stats_part.q. It will show whether it works or not. But my code changes are based on debugger sessions so it should work (I mean multiple inserts). Also if a method like get_partitions_statistics_req does not propagate txnId and write id list then either of the two: 1. it is not used by StatsOptimizer in case SELECT. 2. it is not used in the execution patch for a SELECT. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18729) Druid Time column type
[ https://issues.apache.org/jira/browse/HIVE-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526611#comment-16526611 ] Jesus Camacho Rodriguez commented on HIVE-18729: [~ashutoshc], patch requires release of Calcite 1.17.0. > Druid Time column type > -- > > Key: HIVE-18729 > URL: https://issues.apache.org/jira/browse/HIVE-18729 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Blocker > Attachments: HIVE-18729.01.branch-3.patch, HIVE-18729.branch-3.patch, > HIVE-18729.patch > > > I have talked Offline with [~jcamachorodriguez] about this and agreed that > the best way to go is to support both cases where Druid time column can be > Timestamp or Timestamp with local time zone. > In fact, for the Hive-Druid internal table, this makes perfect sense since we > have Hive metadata about the time column during the CTAS statement then we > can handle both cases as we do for another type of storage eg ORC. > For the Druid external tables, we can have a default type and allow the user > to override that via table properties. > CC [~ashutoshc] and [~nishantbangarwa]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18729) Druid Time column type
[ https://issues.apache.org/jira/browse/HIVE-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526609#comment-16526609 ] Ashutosh Chauhan commented on HIVE-18729: - [~jcamachorodriguez] Looks like this needs patch reattached to trigger a run. > Druid Time column type > -- > > Key: HIVE-18729 > URL: https://issues.apache.org/jira/browse/HIVE-18729 > Project: Hive > Issue Type: Task > Components: Druid integration >Reporter: slim bouguerra >Assignee: Jesus Camacho Rodriguez >Priority: Blocker > Attachments: HIVE-18729.01.branch-3.patch, HIVE-18729.branch-3.patch, > HIVE-18729.patch > > > I have talked Offline with [~jcamachorodriguez] about this and agreed that > the best way to go is to support both cases where Druid time column can be > Timestamp or Timestamp with local time zone. > In fact, for the Hive-Druid internal table, this makes perfect sense since we > have Hive metadata about the time column during the CTAS statement then we > can handle both cases as we do for another type of storage eg ORC. > For the Druid external tables, we can have a default type and allow the user > to override that via table properties. > CC [~ashutoshc] and [~nishantbangarwa]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20026) Create metastore upgrade and schema changes for 3.2.0
[ https://issues.apache.org/jira/browse/HIVE-20026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20026: -- > Create metastore upgrade and schema changes for 3.2.0 > - > > Key: HIVE-20026 > URL: https://issues.apache.org/jira/browse/HIVE-20026 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526593#comment-16526593
]
Aihua Xu commented on HIVE-19668:
-
The patch looks good to me. +1 pending tests.
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, HIVE-19668.02.patch,
> image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
