[jira] [Commented] (HIVE-21102) Optimize SparkPlanGenerator for getInputPaths (emptyFile checks)
[
https://issues.apache.org/jira/browse/HIVE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741010#comment-16741010
]
Hive QA commented on HIVE-21102:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954666/HIVE-21102.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 15694 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[global_limit]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin1]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_stats]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ppd_union_view]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ppr_pushdown]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[smb_mapjoin_18]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dml]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_empty]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=173)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_noscan_2]
(batchId=127)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954666 - PreCommit-HIVE-Build
> Optimize SparkPlanGenerator for getInputPaths (emptyFile checks)
>
>
> Key: HIVE-21102
> URL: https://issues.apache.org/jira/browse/HIVE-21102
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-21102.1.patch, HIVE-21102.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21102) Optimize SparkPlanGenerator for getInputPaths (emptyFile checks)
[
https://issues.apache.org/jira/browse/HIVE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740993#comment-16740993
]
Hive QA commented on HIVE-21102:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15604/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize SparkPlanGenerator for getInputPaths (emptyFile checks)
>
>
> Key: HIVE-21102
> URL: https://issues.apache.org/jira/browse/HIVE-21102
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-21102.1.patch, HIVE-21102.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740988#comment-16740988
]
Hive QA commented on HIVE-20170:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954647/HIVE-20170.10.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15694 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15603/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15603/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15603/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954647 - PreCommit-HIVE-Build
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740981#comment-16740981
]
Hive QA commented on HIVE-20699:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954634/HIVE-20699.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15602/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15602/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15602/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:53.506
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15602/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:53.509
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-12 02:57:54.161
+ rm -rf ../yetus_PreCommit-HIVE-Build-15602
+ mkdir ../yetus_PreCommit-HIVE-Build-15602
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15602
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15602/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not
exist in index
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java: does
not exist in index
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/SplitGrouper.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java:
does not exist in index
error: a/ql/src/test/results/clientpositive/show_functions.q.out: does not
exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java:375
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java' with
conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:12: trailing whitespace.
SPLIT_GROUPING_MODE("hive.split.grouping.mode", "query", new
StringSet("query", "compactor"),
/data/hiveptest/working/scratch/build.patch:33: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:359: trailing whitespace.
/data/hiveptest/working/scratch/build.patch:583: trailing
[jira] [Commented] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740983#comment-16740983
]
Hive QA commented on HIVE-20170:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15603/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15603/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21102) Optimize SparkPlanGenerator for getInputPaths (emptyFile checks)
[ https://issues.apache.org/jira/browse/HIVE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-21102: Attachment: HIVE-21102.2.patch > Optimize SparkPlanGenerator for getInputPaths (emptyFile checks) > > > Key: HIVE-21102 > URL: https://issues.apache.org/jira/browse/HIVE-21102 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-21102.1.patch, HIVE-21102.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740954#comment-16740954
]
Hive QA commented on HIVE-21045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 188 unchanged - 0 fixed = 189 total (was 188) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Should
org.apache.hadoop.hive.metastore.datasource.BoneCPDataSourceProvider$BoneCPMetrics
be a _static_ inner class? At BoneCPDataSourceProvider.java:inner class? At
BoneCPDataSourceProvider.java:[lines 105-166] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15601/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15601/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15601/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use diff
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740951#comment-16740951 ] Vineet Garg commented on HIVE-21107: [~jdere] [~gopalv] Can you take a look please? > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740945#comment-16740945
]
Hive QA commented on HIVE-21107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954632/HIVE-21107.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15695 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15600/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15600/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15600/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954632 - PreCommit-HIVE-Build
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740918#comment-16740918
]
Hive QA commented on HIVE-21107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
33s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 74 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15600/dev-support/hive-personality.sh
|
| git revision | master / 28db173 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15600/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15600/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cannot find field" error during dynamically partitioned hash join
> -
>
> Key: HIVE-21107
> URL: https://issues.apache.org/jira/browse/HIVE-21107
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch,
> HIVE-21107.3.patch
>
>
> This occurs in non-CBO path with dynamic partitioned join + constant
> propagation ON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21113) For HPL/SQL that contains boolean expression with NOT, incorrect SQL may be generated.
[ https://issues.apache.org/jira/browse/HIVE-21113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-21113: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) +1. Patch pushed to master. Thanks for contributing! > For HPL/SQL that contains boolean expression with NOT, incorrect SQL may be > generated. > --- > > Key: HIVE-21113 > URL: https://issues.apache.org/jira/browse/HIVE-21113 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Baoning He >Assignee: Baoning He >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21113.1.patch > > > In HPL/SQL, ' SELECT * FROM a WHERE NOT (1 = 2) ' will generate to incorrect > SQL ' SELECT * FROM a WHERE (1 = 2) ', the 'NOT' in boolean expression is > missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740895#comment-16740895
]
Hive QA commented on HIVE-17020:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954631/HIVE-17020.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 15694 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=278)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket2] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_groupby3_noskew_multi_distinct]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_map]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_map_multi_distinct]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_map_skew]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_noskew]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby3_noskew_multi_distinct]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_map]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_map_skew]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby5_map]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby5_map_skew]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input30] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input32] (batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_gby2]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_mixed]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_union_src]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_orderby]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf3] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_6]
(batchId=15)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[serde_typedbytes4]
(batchId=270)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket2]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket4]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers2]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers_acid]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainanalyze_2]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dml]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_multiinsert]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_6]
(batchId=162)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
(batchId=136)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15599/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15599/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15599/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.p
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Description:
With the attached UDAF the following query crashes on hive.
CRASHES
{noformat}
select rs_max(genderkey),count(distinct genderkey) from as_adventure.dimgender;
{noformat}
WORKS
{noformat}
select rs_max(genderkey) from as_adventure.dimgender;
{noformat}
The table looks like
{noformat}
0: jdbc:hive2://localhost:1> select * from dimgender;
OK
INFO : Compiling
command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
select * from dimgender
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema:
Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
comment:null), FieldSchema(name:dimgender.gendername, type:string,
comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7); Time
taken: 0.2 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing
command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
select * from dimgender
INFO : Completed executing
command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7); Time
taken: 0.004 seconds
INFO : OK
INFO : Concurrency mode is disabled, not creating a lock manager
+--+---+
| dimgender.genderkey | dimgender.gendername |
+--+---+
| M| Male |
| F| Female|
| U| Unisex|
+--+---+
{noformat}
{noformat}
Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
failure ) :
attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=0)
{"key":{"_col0":"F"},"value":{"_col0":"F"}}
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
{noformat}
...
{noformat}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute
method public boolean com.sample.MaxUDA$Evaluator.merge(java.lang.String) with
arguments {F}:argument type mismatch
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
{noformat}
PLAN
{noformat}
++
| Explain |
++
| Plan optimized by CBO. |
||
| Vertex dependency in root stage|
| Reducer 2 <- Map 1 (SIMPLE_EDGE) |
| Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
||
| Stage-0|
| Fetch Operator |
| limit:-1 |
| Stage-1|
| Reducer 3|
| File Output Operator [FS_6] |
| Group By Operator [GBY_12] (rows=1 width=368) |
|
Output:["_col0","_col1"],aggregations:["rs_max(VALUE._col0)","count(VALUE._col1)"]
|
| <-Reducer 2 [CUSTOM_SIMPLE_EDGE] |
| PARTITION_ONLY_SHUFFLE [RS_11] |
| Group By Operator [GBY_10] (rows=1 width=368) |
|
Output:["_col0","_col1"],aggregations:["rs_max(_col1)","count(_col0)"] |
| Group By Operator [GBY_9] (rows=3 width=2) |
|
Output:["_col0","_col1"],aggregations:["rs_max(VALUE._col0)"],keys:KEY._col0 |
| <-Map 1 [SIMPLE_EDGE]|
|
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Attachment: run.log
> String UDAF and count distinct in the same select give error
>
>
> Key: HIVE-21119
> URL: https://issues.apache.org/jira/browse/HIVE-21119
> Project: Hive
> Issue Type: Bug
>Reporter: Ravi Shetye
>Priority: Major
> Labels: wrongresults
> Attachments: MaxUDA.java, run.log
>
>
> With the attached UDAF the following query crashes on hive.
> CRASHES
> {noformat}
> select rs_max(genderkey),count(distinct genderkey) from
> as_adventure.dimgender;
> {noformat}
> WORKS
> {noformat}
> select rs_max(genderkey) from as_adventure.dimgender;
> {noformat}
> The table looks like
> {noformat}
> 0: jdbc:hive2://localhost:1> select * from dimgender;
> OK
> INFO : Compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
> comment:null), FieldSchema(name:dimgender.gendername, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.2 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Completed executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.004 seconds
> INFO : OK
> INFO : Concurrency mode is disabled, not creating a lock manager
> +--+---+
> | dimgender.genderkey | dimgender.gendername |
> +--+---+
> | M| Male |
> | F| Female|
> | U| Unisex|
> +--+---+
> {noformat}
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
> diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"F"},"value":{"_col0":"F"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> {noformat}
> ...
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> execute method public boolean
> com.sample.MaxUDA$Evaluator.merge(java.lang.String) with arguments
> {F}:argument type mismatch
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Attachment: MaxUDA.java
> String UDAF and count distinct in the same select give error
>
>
> Key: HIVE-21119
> URL: https://issues.apache.org/jira/browse/HIVE-21119
> Project: Hive
> Issue Type: Bug
>Reporter: Ravi Shetye
>Priority: Major
> Labels: wrongresults
> Attachments: MaxUDA.java
>
>
> With the attached UDAF the following query crashes on hive.
> CRASHES
> {noformat}
> select rs_max(genderkey),count(distinct genderkey) from
> as_adventure.dimgender;
> {noformat}
> WORKS
> {noformat}
> select rs_max(genderkey) from as_adventure.dimgender;
> {noformat}
> The table looks like
> {noformat}
> 0: jdbc:hive2://localhost:1> select * from dimgender;
> OK
> INFO : Compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
> comment:null), FieldSchema(name:dimgender.gendername, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.2 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Completed executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.004 seconds
> INFO : OK
> INFO : Concurrency mode is disabled, not creating a lock manager
> +--+---+
> | dimgender.genderkey | dimgender.gendername |
> +--+---+
> | M| Male |
> | F| Female|
> | U| Unisex|
> +--+---+
> {noformat}
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
> diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"F"},"value":{"_col0":"F"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> {noformat}
> ...
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> execute method public boolean
> com.sample.MaxUDA$Evaluator.merge(java.lang.String) with arguments
> {F}:argument type mismatch
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740880#comment-16740880
]
Vihang Karajgaonkar commented on HIVE-21115:
Hi [~odraese] are you suggesting the HiveQL transactions. There a couple of
issues with using the writerID. It assumes that all the clients support
transactions and it is enabled. That said I am not super familiar with this
part of the code. Can you point me to the code location of the writerId?
Specifically how does it get generated and when does it get updated?
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, another client which
> wants to do a alter operation has no good way to know if the object which it
> has is the same as the one stored in metastore. Metastore updates the
> {{transient_lastDdlTime}} every time there is a DDL operation on the object.
> However, this value cannot be relied for all the clients since after
> HIVE-1768 metastore updates the value only when it is not set in the
> parameters. It is possible that a client which alters the object state, does
> not remove the {{transient_lastDdlTime}} and metastore will not update it.
> Secondly, if there is a clock skew between multiple HMS instances when HMS-HA
> is configured, time values cannot be relied on to find out the sequence of
> alter operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus
> http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
> generate a incrementing sequence number every time a object is altered. The
> value of this object can be set as one of the values in the parameters. The
> advantage of using Datanucleus the versioning can be done across HMS
> instances as part of the database transaction and it should work for all the
> supported databases.
> In theory such a version can be used to detect if the client is presenting a
> object which is "stale" when issuing a alter request. Metastore can choose to
> reject such a alter request since the client may be caching a old version of
> the object and any alter operation on such stale object can potentially
> overwrite previous operations. However, this is can be done in a separate
> JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740871#comment-16740871
]
Hive QA commented on HIVE-17020:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 10 unchanged - 1
fixed = 10 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 1 new + 2310 unchanged - 0 fixed = 2311
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to numPartAndBuck in
org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.getReduceSinkOp(List,
List, List, List, ArrayList, ArrayList, int, Operator, AcidUtils$Operation)
At
SortedDynPartitionOptimizer.java:org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer$SortedDynamicPartitionProc.getReduceSinkOp(List,
List, List, List, ArrayList, ArrayList, int, Operator, AcidUtils$Operation)
At SortedDynPartitionOptimizer.java:[line 493] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15599/dev-support/hive-personality.sh
|
| git revision | master / 28db173 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15599/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15599/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch, HIVE-17020.6.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
>
[jira] [Commented] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740861#comment-16740861
]
Oliver Draese commented on HIVE-21115:
--
One thing that might be worth considering here is the requirement to include
DDL in general transaction management. If we add support for multiple
statements within a single transaction and allow DDL statements to be part of
these transactions, then we probably have the WriterID of the transaction
already as a version identifier. The WriterID could also be used to detect
optimistic locking conflicts and the WriterID column could then also be used as
JDO version field.
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, another client which
> wants to do a alter operation has no good way to know if the object which it
> has is the same as the one stored in metastore. Metastore updates the
> {{transient_lastDdlTime}} every time there is a DDL operation on the object.
> However, this value cannot be relied for all the clients since after
> HIVE-1768 metastore updates the value only when it is not set in the
> parameters. It is possible that a client which alters the object state, does
> not remove the {{transient_lastDdlTime}} and metastore will not update it.
> Secondly, if there is a clock skew between multiple HMS instances when HMS-HA
> is configured, time values cannot be relied on to find out the sequence of
> alter operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus
> http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
> generate a incrementing sequence number every time a object is altered. The
> value of this object can be set as one of the values in the parameters. The
> advantage of using Datanucleus the versioning can be done across HMS
> instances as part of the database transaction and it should work for all the
> supported databases.
> In theory such a version can be used to detect if the client is presenting a
> object which is "stale" when issuing a alter request. Metastore can choose to
> reject such a alter request since the client may be caching a old version of
> the object and any alter operation on such stale object can potentially
> overwrite previous operations. However, this is can be done in a separate
> JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Status: Patch Available (was: Open)
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Attachment: HIVE-20170.10.patch
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Status: Open (was: Patch Available)
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.10.patch,
> HIVE-20170.2.patch, HIVE-20170.3.patch, HIVE-20170.4.patch,
> HIVE-20170.5.patch, HIVE-20170.6.patch, HIVE-20170.7.patch,
> HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740857#comment-16740857
]
Vihang Karajgaonkar commented on HIVE-21115:
Thanks [~ekoifman] for the suggestion. The only direct SQL path which I know
which modifies the HMS objects is dropTable which probably doesn't apply in
this case since the object is deleted. All the alter calls use JDO to the best
of my knowledge. On-update trigger is interesting idea. How do you envision the
transfer of the updated version value from the trigger to translate to the
thrift object. I think the trigger will execute as part of the transaction so
the updated value will only visible once the transaction commits.
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, another client which
> wants to do a alter operation has no good way to know if the object which it
> has is the same as the one stored in metastore. Metastore updates the
> {{transient_lastDdlTime}} every time there is a DDL operation on the object.
> However, this value cannot be relied for all the clients since after
> HIVE-1768 metastore updates the value only when it is not set in the
> parameters. It is possible that a client which alters the object state, does
> not remove the {{transient_lastDdlTime}} and metastore will not update it.
> Secondly, if there is a clock skew between multiple HMS instances when HMS-HA
> is configured, time values cannot be relied on to find out the sequence of
> alter operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus
> http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
> generate a incrementing sequence number every time a object is altered. The
> value of this object can be set as one of the values in the parameters. The
> advantage of using Datanucleus the versioning can be done across HMS
> instances as part of the database transaction and it should work for all the
> supported databases.
> In theory such a version can be used to detect if the client is presenting a
> object which is "stale" when issuing a alter request. Metastore can choose to
> reject such a alter request since the client may be caching a old version of
> the object and any alter operation on such stale object can potentially
> overwrite previous operations. However, this is can be done in a separate
> JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21002) Backwards incompatible change: Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x incorrectly
[
https://issues.apache.org/jira/browse/HIVE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740856#comment-16740856
]
Owen O'Malley commented on HIVE-21002:
--
The behavior of Avro and Parquet is wrong both in 2.x and 3.1. The path forward
should be to match the desired Hive semantics and return '00:00:00' for new
files, regardless of format.
Iceberg uses Parquet's isAdjustedToUTC = true for timestamptz, which is the
equivalent of Hive's timestamp with local time zone and isAdjustedToUTC = false
for timestamp. It would be good to match those semantics in Hive. Can we detect
the version of Hive that wrote the Parquet file to provide compatibility with
told files?
> Backwards incompatible change: Hive 3.1 reads back Avro and Parquet
> timestamps written by Hive 2.x incorrectly
> --
>
> Key: HIVE-21002
> URL: https://issues.apache.org/jira/browse/HIVE-21002
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Zoltan Ivanfi
>Priority: Major
>
> Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x
> incorrectly. As an example session to demonstrate this problem, create a
> dataset using Hive version 2.x in America/Los_Angeles:
> {code:sql}
> hive> create table ts_‹format› (ts timestamp) stored as ‹format›;
> hive> insert into ts_‹format› values (*‘2018-01-01 00:00:00.000’*);
> {code}
> Querying this table by issuing
> {code:sql}
> hive> select * from ts_‹format›;
> {code}
> from different time zones using different versions of Hive and different
> storage formats gives the following results:
> |‹format›|Time zone|Hive 2.x|Hive 3.1|
> |Avro and Parquet|America/Los_Angeles|2018-01-01 *00*:00:00.0|2018-01-01
> *08*:00:00.0|
> |Avro and Parquet|Europe/Paris|2018-01-01 *09*:00:00.0|2018-01-01
> *08*:00:00.0|
> |Textfile and ORC|America/Los_Angeles|2018-01-01 00:00:00.0|2018-01-01
> 00:00:00.0|
> |Textfile and ORC|Europe/Paris|2018-01-01 00:00:00.0|2018-01-01 00:00:00.0|
> *Hive 3.1 clearly gives different results than Hive 2.x for timestamps stored
> in Avro and Parquet formats.* Apache ORC behaviour has not changed because it
> was modified to adjust timestamps to retain backwards compatibility. Textfile
> behaviour has not changed, because its processing involves parsing and
> formatting instead of proper serializing and deserializing, so they
> inherently had LocalDateTime semantics even in Hive 2.x.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21002) Backwards incompatible change: Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x incorrectly
[
https://issues.apache.org/jira/browse/HIVE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740856#comment-16740856
]
Owen O'Malley edited comment on HIVE-21002 at 1/11/19 10:53 PM:
The behavior of Avro and Parquet is wrong both in 2.x and 3.1. The path forward
should be to match the desired Hive semantics and return '00:00:00' for new
files, regardless of format.
Iceberg uses Parquet's isAdjustedToUTC = true for timestamptz, which is the
equivalent of Hive's timestamp with local time zone and isAdjustedToUTC = false
for timestamp. It would be good to match those semantics in Hive. Can we detect
the version of Hive that wrote the Parquet file to provide compatibility with
old files?
was (Author: owen.omalley):
The behavior of Avro and Parquet is wrong both in 2.x and 3.1. The path forward
should be to match the desired Hive semantics and return '00:00:00' for new
files, regardless of format.
Iceberg uses Parquet's isAdjustedToUTC = true for timestamptz, which is the
equivalent of Hive's timestamp with local time zone and isAdjustedToUTC = false
for timestamp. It would be good to match those semantics in Hive. Can we detect
the version of Hive that wrote the Parquet file to provide compatibility with
told files?
> Backwards incompatible change: Hive 3.1 reads back Avro and Parquet
> timestamps written by Hive 2.x incorrectly
> --
>
> Key: HIVE-21002
> URL: https://issues.apache.org/jira/browse/HIVE-21002
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.1.1
>Reporter: Zoltan Ivanfi
>Priority: Major
>
> Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x
> incorrectly. As an example session to demonstrate this problem, create a
> dataset using Hive version 2.x in America/Los_Angeles:
> {code:sql}
> hive> create table ts_‹format› (ts timestamp) stored as ‹format›;
> hive> insert into ts_‹format› values (*‘2018-01-01 00:00:00.000’*);
> {code}
> Querying this table by issuing
> {code:sql}
> hive> select * from ts_‹format›;
> {code}
> from different time zones using different versions of Hive and different
> storage formats gives the following results:
> |‹format›|Time zone|Hive 2.x|Hive 3.1|
> |Avro and Parquet|America/Los_Angeles|2018-01-01 *00*:00:00.0|2018-01-01
> *08*:00:00.0|
> |Avro and Parquet|Europe/Paris|2018-01-01 *09*:00:00.0|2018-01-01
> *08*:00:00.0|
> |Textfile and ORC|America/Los_Angeles|2018-01-01 00:00:00.0|2018-01-01
> 00:00:00.0|
> |Textfile and ORC|Europe/Paris|2018-01-01 00:00:00.0|2018-01-01 00:00:00.0|
> *Hive 3.1 clearly gives different results than Hive 2.x for timestamps stored
> in Avro and Parquet formats.* Apache ORC behaviour has not changed because it
> was modified to adjust timestamps to retain backwards compatibility. Textfile
> behaviour has not changed, because its processing involves parsing and
> formatting instead of proper serializing and deserializing, so they
> inherently had LocalDateTime semantics even in Hive 2.x.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740854#comment-16740854
]
Hive QA commented on HIVE-20170:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954625/HIVE-20170.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15694 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_memcheck]
(batchId=45)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15598/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15598/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15598/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954625 - PreCommit-HIVE-Build
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.2.patch,
> HIVE-20170.3.patch, HIVE-20170.4.patch, HIVE-20170.5.patch,
> HIVE-20170.6.patch, HIVE-20170.7.patch, HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-8133) Support Postgres via DirectSQL
[ https://issues.apache.org/jira/browse/HIVE-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740839#comment-16740839 ] Karthik Manamcheri commented on HIVE-8133: -- [~damien.carol] [~brocknoland] Is this JIRA even relevant anymore? There is no description. Postgres DirectSQL is supported in Hive 2.x releases, right? If so, can we close this ticket as Done? > Support Postgres via DirectSQL > -- > > Key: HIVE-8133 > URL: https://issues.apache.org/jira/browse/HIVE-8133 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740832#comment-16740832
]
Hive QA commented on HIVE-20170:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15598/dev-support/hive-personality.sh
|
| git revision | master / cb9d5cc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15598/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.2.patch,
> HIVE-20170.3.patch, HIVE-20170.4.patch, HIVE-20170.5.patch,
> HIVE-20170.6.patch, HIVE-20170.7.patch, HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740825#comment-16740825
]
Deepak Jaiswal commented on HIVE-16976:
---
Committed to master.
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.10.patch,
> HIVE-16976.11.patxh, HIVE-16976.2.patch, HIVE-16976.3.patch,
> HIVE-16976.4.patch, HIVE-16976.5.patch, HIVE-16976.6.patch,
> HIVE-16976.7.patch, HIVE-16976.8.patch, HIVE-16976.9.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Reducer 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized, llap
> File Output Operator [FS_36]
> Group By Operator [GBY_35] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_34]
> Group By Operator [GBY_33] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(1)"]
> Select Operator [SEL_32] (rows=9600142089 width=16)
> Filter Operator [FIL_31] (rows=9600142089 width=16)
> predicate:(_col0 > _col1)
> Map Join Operator [MAPJOIN_30] (rows=28800426268 width=16)
> Conds:(Inner),Output:["_col0","_col1"]
> <-Reducer 4 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_28]
> Group By Operator [GBY_27] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 3 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_26]
> Group By Operator [GBY_25] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(d_date_sk)"]
> Select Operator [SEL_24] (rows=652 width=12)
> Output:["d_date_sk"]
> Filter Operator [FIL_23] (rows=652 width=12)
> predicate:(d_year = 2017)
> TableScan [TS_2] (rows=73049 width=12)
>
> tpcds_bin_partitioned_newschema_orc_1@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE,Output:["d_date_sk","d_year"]
> <-Select Operator [SEL_29] (rows=28800426268 width=8)
> Output:["_col0"]
> TableScan [TS_0] (rows=28800426268 width=172)
>
> tpcds_bin_partitioned_newschema_orc_1@store_sales,store_sales,Tbl:COMPLETE,Col:COMPLETE
> {code}
> The SyntheticJoinPredicate is only injected for equi joins, not for < or >
> scalar subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20980) Reinstate Parquet timestamp conversion between HS2 time zone and UTC
[ https://issues.apache.org/jira/browse/HIVE-20980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740823#comment-16740823 ] Jesus Camacho Rodriguez commented on HIVE-20980: Apparently there is a UTC parameter for the timestamp type already in Parquet: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp That should make things much easier. > Reinstate Parquet timestamp conversion between HS2 time zone and UTC > > > Key: HIVE-20980 > URL: https://issues.apache.org/jira/browse/HIVE-20980 > Project: Hive > Issue Type: Sub-task > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-20980.1.patch, HIVE-20980.2.patch, > HIVE-20980.2.patch > > > With HIVE-20007, Parquet timestamps became timezone-agnostic. This means that > timestamps written after the change are read exactly as they were written; > but timestamps stored before this change are effectively converted from the > writing HS2 server time zone to GMT time zone. This patch reinstates the > original behavior: timestamps are converted to UTC before write and from UTC > before read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21057) Make semi join reduction work with UNION
[ https://issues.apache.org/jira/browse/HIVE-21057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740816#comment-16740816 ] Vineet Garg commented on HIVE-21057: [~t3rmin4t0r] [~jcamachorodriguez] Would you mind taking a look at this? This change patches the logic in task generation. Now that there could be pattern where along with UNION having multiple incoming branches it can also have multiple outgoing branches (due to Semi join reduction). This was causing task generation to skip generating a vertex corresponding to one branch. The patch is fixing the logic take care of this case. > Make semi join reduction work with UNION > > > Key: HIVE-21057 > URL: https://issues.apache.org/jira/browse/HIVE-21057 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21057.1.patch, HIVE-21057.2.patch, > HIVE-21057.3.patch > > > HIVE-21007 disabled semi-join for plans containing UNION since it generates a > operator tree which ends up producing invalid tez plan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740809#comment-16740809
]
Hive QA commented on HIVE-20847:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954623/HIVE-20847.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15713 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=243)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15597/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15597/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15597/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954623 - PreCommit-HIVE-Build
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740806#comment-16740806 ] Gopal V commented on HIVE-20801: [~ekoifman]: The ETL is via Spark and Sqoop, so the reads aren't consistent right now. Just that Hive is slowed down by a magnitude when ACID is enabled for even 1 tables. There is no read consistency when files are being modified outside of hive on HDFS. > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-20699:
Attachment: HIVE-20699.5.patch
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch, HIVE-20699.5.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20960) Make MM compactor run in a transaction and remove CompactorMR.createCompactorMarker()
[
https://issues.apache.org/jira/browse/HIVE-20960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-20960:
--
Resolution: Fixed
Fix Version/s: 4.0.0
Release Note: n/a
Status: Resolved (was: Patch Available)
committed to master
thanks Vaibhav for the review
> Make MM compactor run in a transaction and remove
> CompactorMR.createCompactorMarker()
> -
>
> Key: HIVE-20960
> URL: https://issues.apache.org/jira/browse/HIVE-20960
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20960.01.patch, HIVE-20960.02.patch,
> HIVE-20960.03.patch, HIVE-20960.04.patch, HIVE-20960.05.patch,
> HIVE-20960.06.patch, HIVE-20960.07.patch
>
>
> Now that we have HIVE-20823, we know if a dir is produced by compactor from
> the name and {{CompactorMR.createCompactorMarker()}} can be removed.
>
> Also includes a fix to insert_only (MM tables) ACID table so that compactor
> produced base_X has the base_X_vY format as for full CRUD tables
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740793#comment-16740793
]
Eugene Koifman commented on HIVE-21115:
---
Isn't there a direct SQL path somewhere that modifies HMS objects w/o using
DataNucleus?
Could this be expressed via some on-update trigger instead?
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, another client which
> wants to do a alter operation has no good way to know if the object which it
> has is the same as the one stored in metastore. Metastore updates the
> {{transient_lastDdlTime}} every time there is a DDL operation on the object.
> However, this value cannot be relied for all the clients since after
> HIVE-1768 metastore updates the value only when it is not set in the
> parameters. It is possible that a client which alters the object state, does
> not remove the {{transient_lastDdlTime}} and metastore will not update it.
> Secondly, if there is a clock skew between multiple HMS instances when HMS-HA
> is configured, time values cannot be relied on to find out the sequence of
> alter operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus
> http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
> generate a incrementing sequence number every time a object is altered. The
> value of this object can be set as one of the values in the parameters. The
> advantage of using Datanucleus the versioning can be done across HMS
> instances as part of the database transaction and it should work for all the
> supported databases.
> In theory such a version can be used to detect if the client is presenting a
> object which is "stale" when issuing a alter request. Metastore can choose to
> reject such a alter request since the client may be caching a old version of
> the object and any alter operation on such stale object can potentially
> overwrite previous operations. However, this is can be done in a separate
> JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740788#comment-16740788 ] Eugene Koifman commented on HIVE-20801: --- [~gopalv] what do you mean locks are advisory? By default, (with Acid on), non-transactional tables use standard S/X locks so so assuming things are configured properly (and no partial failures), reads should be consistent. It seems that the description of the property is misleading. Also, if you are disabling all locks for readers, why acquire any locks for writers? > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Manamcheri updated HIVE-21045:
--
Attachment: HIVE-21045.3.patch
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch,
> HIVE-21045.3.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740773#comment-16740773
]
Hive QA commented on HIVE-20847:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 19
fixed = 0 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15597/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15597/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740760#comment-16740760
]
Vaibhav Gumashta commented on HIVE-20699:
-
The failed test is due to the new validate_acid_sort_order udf function not
showing up in the.out for show functions. Fixing the output.
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20960) Make MM compactor run in a transaction and remove CompactorMR.createCompactorMarker()
[
https://issues.apache.org/jira/browse/HIVE-20960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740750#comment-16740750
]
Vaibhav Gumashta commented on HIVE-20960:
-
+1
> Make MM compactor run in a transaction and remove
> CompactorMR.createCompactorMarker()
> -
>
> Key: HIVE-20960
> URL: https://issues.apache.org/jira/browse/HIVE-20960
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20960.01.patch, HIVE-20960.02.patch,
> HIVE-20960.03.patch, HIVE-20960.04.patch, HIVE-20960.05.patch,
> HIVE-20960.06.patch, HIVE-20960.07.patch
>
>
> Now that we have HIVE-20823, we know if a dir is produced by compactor from
> the name and {{CompactorMR.createCompactorMarker()}} can be removed.
>
> Also includes a fix to insert_only (MM tables) ACID table so that compactor
> produced base_X has the base_X_vY format as for full CRUD tables
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740747#comment-16740747
]
Hive QA commented on HIVE-20699:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954613/HIVE-20699.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15716 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions]
(batchId=79)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15596/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15596/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15596/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954613 - PreCommit-HIVE-Build
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Open (was: Patch Available)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch, HIVE-17020.6.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Status: Open (was: Patch Available) > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Attachment: HIVE-21107.3.patch > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21107: --- Status: Patch Available (was: Open) > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740736#comment-16740736
]
Vineet Garg commented on HIVE-17020:
[~lirui] Failure wasn't just due to output files. There were couple of issues
with deduplication logic as well as SPDO logic.
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch, HIVE-17020.6.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Status: Patch Available (was: Open)
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch, HIVE-17020.6.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
[
https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-17020:
---
Attachment: HIVE-17020.6.patch
> Aggressive RS dedup can incorrectly remove OP tree branch
> -
>
> Key: HIVE-17020
> URL: https://issues.apache.org/jira/browse/HIVE-17020
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17020.1.patch, HIVE-17020.2.patch,
> HIVE-17020.3.patch, HIVE-17020.4.patch, HIVE-17020.5.patch, HIVE-17020.6.patch
>
>
> Suppose we have an OP tree like this:
> {noformat}
> ...
> |
> RS[1]
> |
> SEL[2]
> /\
> SEL[3] SEL[4]
> | |
> RS[5] FS[6]
> |
> ...
> {noformat}
> When doing aggressive RS dedup, we'll remove all the operators between RS5
> and RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740714#comment-16740714
]
Hive QA commented on HIVE-20699:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 18 new + 512 unchanged - 1
fixed = 530 total (was 513) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 24 new + 184
unchanged - 0 fixed = 208 total (was 184) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
57s{color} | {color:red} ql generated 2 new + 2309 unchanged - 0 fixed = 2311
total (was 2309) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
53s{color} | {color:red} ql generated 3 new + 97 unchanged - 3 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.buildCrudMajorCompactionQuery(HiveConf,
Table, Partition, String) concatenates strings using + in a loop At
CompactorMR.java:strings using + in a loop At CompactorMR.java:[line 530] |
| |
org.apache.hadoop.hive.ql.udf.generic.GenericUDFValidateAcidSortOrder$WriteIdRowId
defines compareTo(GenericUDFValidateAcidSortOrder$WriteIdRowId) and uses
Object.equals() At GenericUDFValidateAcidSortOrder.java:Object.equals() At
GenericUDFValidateAcidSortOrder.java:[lines 85-91] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15596/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15596/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15596/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15596
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21052: -- Affects Version/s: (was: 3.1.1) 3.0.0 > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21118) Make sure if dynamic partitions is true only there's only one writeId allocated
[ https://issues.apache.org/jira/browse/HIVE-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21118: -- Component/s: Transactions > Make sure if dynamic partitions is true only there's only one writeId > allocated > --- > > Key: HIVE-21118 > URL: https://issues.apache.org/jira/browse/HIVE-21118 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > > See > https://issues.apache.org/jira/browse/HIVE-21052?focusedCommentId=16740528&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16740528 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Status: Open (was: Patch Available)
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Status: Open (was: Patch Available)
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.2.patch,
> HIVE-20170.3.patch, HIVE-20170.4.patch, HIVE-20170.5.patch,
> HIVE-20170.6.patch, HIVE-20170.7.patch, HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Attachment: HIVE-20170.9.patch
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.2.patch,
> HIVE-20170.3.patch, HIVE-20170.4.patch, HIVE-20170.5.patch,
> HIVE-20170.6.patch, HIVE-20170.7.patch, HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20170) Improve JoinOperator "rows for join key" Logging
[
https://issues.apache.org/jira/browse/HIVE-20170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20170:
---
Status: Patch Available (was: Open)
Come on... unit tests!
> Improve JoinOperator "rows for join key" Logging
>
>
> Key: HIVE-20170
> URL: https://issues.apache.org/jira/browse/HIVE-20170
> Project: Hive
> Issue Type: Improvement
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20170.1.patch, HIVE-20170.2.patch,
> HIVE-20170.3.patch, HIVE-20170.4.patch, HIVE-20170.5.patch,
> HIVE-20170.6.patch, HIVE-20170.7.patch, HIVE-20170.8.patch, HIVE-20170.9.patch
>
>
> {code:java}
> 2018-06-25 09:37:33,193 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5728000 rows
> for join key [333, 22]
> 2018-06-25 09:37:33,901 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5828000 rows
> for join key [333, 22]
> 2018-06-25 09:37:34,623 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 5928000 rows
> for join key [333, 22]
> 2018-06-25 09:37:35,342 INFO [main]
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 6028000 rows
> for join key [333, 22]
> {code}
> [https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java#L120]
> This logging should use the same facilities as the other Operators for
> emitting this type of log message. HIVE-10078
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Status: Patch Available (was: Open)
Attaching same patch again. Hopefully unit tests pass.
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Attachment: HIVE-20847.3.patch
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Attachment: (was: HIVE-20847.3.patch)
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Attachment: HIVE-20847.3.patch
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 4.0.0, 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch, HIVE-20847.1.patch,
> HIVE-20847.1.patch, HIVE-20847.2.patch, HIVE-20847.3.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20699) Query based compactor for full CRUD Acid tables
[
https://issues.apache.org/jira/browse/HIVE-20699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-20699:
Attachment: HIVE-20699.4.patch
> Query based compactor for full CRUD Acid tables
> ---
>
> Key: HIVE-20699
> URL: https://issues.apache.org/jira/browse/HIVE-20699
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-20699.1.patch, HIVE-20699.1.patch,
> HIVE-20699.2.patch, HIVE-20699.3.patch, HIVE-20699.4.patch
>
>
> Currently the Acid compactor is implemented as generated MR job
> ({{CompactorMR.java}}).
> It could also be expressed as a Hive query that reads from a given partition
> and writes data back to the same partition. This will merge the deltas and
> 'apply' the delete events. The simplest would be to just use Insert
> Overwrite but that will change all ROW__IDs which we don't want.
> Need to implement this in a way that preserves ROW__IDs and creates a new
> {{base_x}} directory to handle Major compaction.
> Minor compaction will be investigated separately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20980) Reinstate Parquet timestamp conversion between HS2 time zone and UTC
[
https://issues.apache.org/jira/browse/HIVE-20980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740613#comment-16740613
]
Jesus Camacho Rodriguez commented on HIVE-20980:
[~klcopp], thanks for your patch. I do not think we have reached an agreement
on how to move forward concerning timestamp types (at least in Hive). What I do
not like about the current proposal is that we still have different semantics
(local date time vs instant) for the same type (timestamp) depending on the
storage format used for the table (e.g text/orc vs parquet/avro). This seems
quite shaky moving forward. In turn, I already described my concerns about
having 4 types where 'timestamp without time zone' does not have same semantics
as 'timestamp'.
The patch is reimplementing 'timestamp with local time zone' semantics into a
'timestamp' and it even relies on the session time zone instead of relying on
the system time zone, which is something that was not done before AFAIK.
Instead of following that path, is it possible to provide an upgrade path from
< 3.x to 3.x where the column type for tables stored using Parquet is altered
when we upgrade ('timestamp' -> 'timestamp with local time zone')? Then Parquet
writer/reader can choose how to store the 'timestamp with local time zone' type
internally, e.g., if it wants to remain compatible with legacy readers, it
could choose to store it as a timestamp. That would provide consistent
semantics moving forward as well as backwards compatibility, albeit DDL
statements created before version 3.x will need to be modified if instant
semantics are required. Is that reasonable? Is there anything I am missing?
Cc [~zi] [~owen.omalley]
> Reinstate Parquet timestamp conversion between HS2 time zone and UTC
>
>
> Key: HIVE-20980
> URL: https://issues.apache.org/jira/browse/HIVE-20980
> Project: Hive
> Issue Type: Sub-task
> Components: File Formats
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-20980.1.patch, HIVE-20980.2.patch,
> HIVE-20980.2.patch
>
>
> With HIVE-20007, Parquet timestamps became timezone-agnostic. This means that
> timestamps written after the change are read exactly as they were written;
> but timestamps stored before this change are effectively converted from the
> writing HS2 server time zone to GMT time zone. This patch reinstates the
> original behavior: timestamps are converted to UTC before write and from UTC
> before read.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16976) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
[
https://issues.apache.org/jira/browse/HIVE-16976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740608#comment-16740608
]
Jesus Camacho Rodriguez commented on HIVE-16976:
[~djaiswal], I left a minor comment in the RB.
+1 (pending tests)
> DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN
>
>
> Key: HIVE-16976
> URL: https://issues.apache.org/jira/browse/HIVE-16976
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.1.1, 3.0.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-16976.1.patch, HIVE-16976.10.patch,
> HIVE-16976.11.patxh, HIVE-16976.2.patch, HIVE-16976.3.patch,
> HIVE-16976.4.patch, HIVE-16976.5.patch, HIVE-16976.6.patch,
> HIVE-16976.7.patch, HIVE-16976.8.patch, HIVE-16976.9.patch
>
>
> Tez DPP does not kick in for scenarios where a user wants to run a comparison
> clause instead of a JOIN/IN clause.
> {code}
> explain select count(1) from store_sales where ss_sold_date_sk > (select
> max(d_Date_sk) from date_dim where d_year = 2017);
> Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
> OK
> Plan optimized by CBO.
> Vertex dependency in root stage
> Map 1 <- Reducer 4 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
> Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized, llap
> File Output Operator [FS_36]
> Group By Operator [GBY_35] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(VALUE._col0)"]
> <-Map 1 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_34]
> Group By Operator [GBY_33] (rows=1 width=8)
> Output:["_col0"],aggregations:["count(1)"]
> Select Operator [SEL_32] (rows=9600142089 width=16)
> Filter Operator [FIL_31] (rows=9600142089 width=16)
> predicate:(_col0 > _col1)
> Map Join Operator [MAPJOIN_30] (rows=28800426268 width=16)
> Conds:(Inner),Output:["_col0","_col1"]
> <-Reducer 4 [BROADCAST_EDGE] vectorized, llap
> BROADCAST [RS_28]
> Group By Operator [GBY_27] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(VALUE._col0)"]
> <-Map 3 [CUSTOM_SIMPLE_EDGE] vectorized, llap
> PARTITION_ONLY_SHUFFLE [RS_26]
> Group By Operator [GBY_25] (rows=1 width=8)
> Output:["_col0"],aggregations:["max(d_date_sk)"]
> Select Operator [SEL_24] (rows=652 width=12)
> Output:["d_date_sk"]
> Filter Operator [FIL_23] (rows=652 width=12)
> predicate:(d_year = 2017)
> TableScan [TS_2] (rows=73049 width=12)
>
> tpcds_bin_partitioned_newschema_orc_1@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE,Output:["d_date_sk","d_year"]
> <-Select Operator [SEL_29] (rows=28800426268 width=8)
> Output:["_col0"]
> TableScan [TS_0] (rows=28800426268 width=172)
>
> tpcds_bin_partitioned_newschema_orc_1@store_sales,store_sales,Tbl:COMPLETE,Col:COMPLETE
> {code}
> The SyntheticJoinPredicate is only injected for equi joins, not for < or >
> scalar subqueries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740585#comment-16740585
]
Hive QA commented on HIVE-21045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 188 unchanged - 0 fixed = 189 total (was 188) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Should
org.apache.hadoop.hive.metastore.datasource.BoneCPDataSourceProvider$BoneCPMetrics
be a _static_ inner class? At BoneCPDataSourceProvider.java:inner class? At
BoneCPDataSourceProvider.java:[lines 105-166] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15595/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15595/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15595/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection poolin
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740569#comment-16740569
]
Hive QA commented on HIVE-21095:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954606/HIVE-21095.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15712 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15594/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15594/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15594/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954606 - PreCommit-HIVE-Build
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch, HIVE-21095.4.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740548#comment-16740548
]
Hive QA commented on HIVE-21095:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15594/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15594/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch, HIVE-21095.4.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
>
[jira] [Updated] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Manamcheri updated HIVE-21045:
--
Attachment: HIVE-21045.2.patch
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Attachments: HIVE-21045.1.patch, HIVE-21045.2.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740538#comment-16740538 ] Jaume M commented on HIVE-21052: Got it, I've opened HIVE-21118 for this. > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21118) Make sure if dynamic partitions is true only there's only one writeId allocated
[ https://issues.apache.org/jira/browse/HIVE-21118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M reassigned HIVE-21118: -- Assignee: Jaume M > Make sure if dynamic partitions is true only there's only one writeId > allocated > --- > > Key: HIVE-21118 > URL: https://issues.apache.org/jira/browse/HIVE-21118 > Project: Hive > Issue Type: Improvement >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > > See > https://issues.apache.org/jira/browse/HIVE-21052?focusedCommentId=16740528&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16740528 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740528#comment-16740528 ] Eugene Koifman commented on HIVE-21052: --- we should never allocate > 1 writeId per (table,txn). That is done somewhere in DbTxnHandler.getTableWriteId(). (perhaps it should also be checked in TxnHandler.allocateTableWriteIds() but I'd do it in a separate jira) Though the stmt/operation being executed may target > 1 table, so there there some minimal processing to look at all LockComponent entries to come up with a set of unique (table,txnid,writeid) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21095:
-
Attachment: HIVE-21095.4.patch
Status: Patch Available (was: Open)
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch, HIVE-21095.4.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21095:
-
Status: Open (was: Patch Available)
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20980) Reinstate Parquet timestamp conversion between HS2 time zone and UTC
[ https://issues.apache.org/jira/browse/HIVE-20980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740513#comment-16740513 ] Karen Coppage commented on HIVE-20980: -- [~jcamachorodriguez], may I have your two cents' worth on this patch? Does it need more discussion with the community or is it ready to go? > Reinstate Parquet timestamp conversion between HS2 time zone and UTC > > > Key: HIVE-20980 > URL: https://issues.apache.org/jira/browse/HIVE-20980 > Project: Hive > Issue Type: Sub-task > Components: File Formats >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-20980.1.patch, HIVE-20980.2.patch, > HIVE-20980.2.patch > > > With HIVE-20007, Parquet timestamps became timezone-agnostic. This means that > timestamps written after the change are read exactly as they were written; > but timestamps stored before this change are effectively converted from the > writing HS2 server time zone to GMT time zone. This patch reinstates the > original behavior: timestamps are converted to UTC before write and from UTC > before read. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21117) A day may belong to a different year than the week it is a part of
[
https://issues.apache.org/jira/browse/HIVE-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740495#comment-16740495
]
Matthias commented on HIVE-21117:
-
The issue can not only happen into one direction, but into the other direction
too. Here a rather simple query that demonstrates this:
{code:java}
SELECT
c.createddatetime,
year(c.createddatetime) as created_year,
month(c.createddatetime) as created_month,
weekofyear(c.createddatetime) as created_week,
if (weekofyear(c.createddatetime) = 1, TRUE, FALSE) as first_week,
if (weekofyear(c.createddatetime) > 50 AND MONTH(c.createddatetime) = 1,
TRUE, FALSE) as last_week,
if (weekofyear(c.createddatetime) = 1 AND MONTH(c.createddatetime) = 12,
year(c.createddatetime) + 1, IF(weekofyear(c.createddatetime) > 50 AND
MONTH(c.createddatetime) = 1, year(c.createddatetime) - 1,
year(c.createddatetime))) as year_of_week
FROM (
SELECT CAST('2018-12-31 17:37:38' AS TIMESTAMP) as createddatetime
UNION
SELECT CAST('2017-01-01 17:37:38' AS TIMESTAMP) as createddatetime
) as c
;
{code}
The last column is a way on how to work around the issue, here the output:
!image-2019-01-11-16-41-27-357.png!
> A day may belong to a different year than the week it is a part of
> --
>
> Key: HIVE-21117
> URL: https://issues.apache.org/jira/browse/HIVE-21117
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 2.3.4
>Reporter: Zoltan Ivanfi
>Priority: Major
>
> When using the year() and weekofyear() functions in a query, their result is
> 2018 and 1 (respectively) for the day '2018-12-31'.
> The year() function returns 2018 for the input '2018-12-31', because that day
> belongs to the year 2018.
> The weekofyear() functions returns 1 for the input '2018-12-31', because that
> day belongs to the first week of 2019.
> Both functions provide sensible results on their own, but when combined, the
> result is wrong, because '2018-12-31' does not belong to week 1 of 2018.
> I suggest adding a new function yearofweek() that would return 2019 for the
> input '2018-12-31' and adding a warning to the documentation of the
> weekofyear() function about this problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740498#comment-16740498
]
Hive QA commented on HIVE-21095:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954601/HIVE-21095.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15593/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15593/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15593/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-11 15:40:58.019
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15593/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-11 15:40:58.022
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f713140 HIVE-21036 extend OpenTxnRequest with transaction type
(Igor Kryvenko via Eugene Koifman)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f713140 HIVE-21036 extend OpenTxnRequest with transaction type
(Igor Kryvenko via Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-11 15:40:58.998
+ rm -rf ../yetus_PreCommit-HIVE-Build-15593
+ mkdir ../yetus_PreCommit-HIVE-Build-15593
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15593
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15593/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:30: trailing whitespace.
tz timestamp with local time zone
/data/hiveptest/working/scratch/build.patch:39: trailing whitespace.
tz1 timestamp with local time zone
warning: 2 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc670848821385867120.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc670848821385867120.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc4138762858751902920.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
log4j:WARN No appenders could be found for logger (DataNucleus.Persistence).
log4j:WARN Please initialize the log4j system properly.
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer completed with success for 41 classe
[jira] [Updated] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21094: - Attachment: (was: HIVE-21094.1.patch) > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21094: - Attachment: (was: HIVE-21094.1.patch) > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21094: - Status: Open (was: Patch Available) > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch, HIVE-21094.1.patch, > HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21094: - Attachment: (was: HIVE-21094.1.patch) > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch, HIVE-21094.1.patch, > HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-21094: - Attachment: (was: HIVE-21094.1.patch) > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch, HIVE-21094.1.patch, > HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21095:
-
Attachment: HIVE-21095.3.patch
Status: Patch Available (was: Open)
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-21095:
-
Status: Open (was: Patch Available)
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740493#comment-16740493
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954585/HIVE-21001.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 144 failed/errored test(s), 15712 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when]
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_to_null]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[folder_predicate]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[in_typecheck_char]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[in_typecheck_mixed]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_6]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_7]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_8]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_6]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join_filter]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udf_case]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rand_partitionpruner3]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp_ints_casts]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_isops_simplify]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_date_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_math_funcs]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_6]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_timestamp_ints_casts]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_duplicate]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=195)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[kryo]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_2]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_null_agg]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_case_when_1]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_date_1]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_math_funcs]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_2]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_6]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_timestamp_ints_casts]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[auto_sortmerge
[jira] [Commented] (HIVE-21045) Add HMS total api count stats and connection pool stats to metrics
[
https://issues.apache.org/jira/browse/HIVE-21045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740465#comment-16740465
]
Yongzhi Chen commented on HIVE-21045:
-
The patch looks good. +1 pending tests.
> Add HMS total api count stats and connection pool stats to metrics
> --
>
> Key: HIVE-21045
> URL: https://issues.apache.org/jira/browse/HIVE-21045
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
> Attachments: HIVE-21045.1.patch
>
>
> There are two key metrics which I think we lack and which would be really
> great to help with scaling visibility in HMS.
> *Total API calls duration stats*
> We already compute and log the duration of API calls in the {{PerfLogger}}.
> We don't have any gauge or timer on what the average duration of an API call
> is for the past some bucket of time. This will give us an insight into if
> there is load on the server which is increasing the average API response time.
>
> *Connection Pool stats*
> We can use different connection pooling libraries such as bonecp or hikaricp.
> These pool managers expose statistics such as average time waiting to get a
> connection, number of connections active, etc. We should expose this as a
> metric so that we can track if the the connection pool size configured is too
> small and we are saturating!
> These metrics would help catch problems with HMS resource contention before
> they actually have jobs failing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.18
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740473#comment-16740473
]
Hive QA commented on HIVE-21001:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15592/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15592/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to calcite-1.18
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch,
> HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch,
> HIVE-21001.05.patch, HIVE-21001.06.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21117) A day may belong to a different year than the week it is a part of
[ https://issues.apache.org/jira/browse/HIVE-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Ivanfi updated HIVE-21117: - Description: When using the year() and weekofyear() functions in a query, their result is 2018 and 1 (respectively) for the day '2018-12-31'. The year() function returns 2018 for the input '2018-12-31', because that day belongs to the year 2018. The weekofyear() functions returns 1 for the input '2018-12-31', because that day belongs to the first week of 2019. Both functions provide sensible results on their own, but when combined, the result is wrong, because '2018-12-31' does not belong to week 1 of 2018. I suggest adding a new function yearofweek() that would return 2019 for the input '2018-12-31' and adding a warning to the documentation of the weekofyear() function about this problem. was: When using the year() and weekofyear() functions in a query, their result is 2018 and 1 (respectively) for the day '2018-12-31'. The year() function returns 2018 for the input '2018-12-31', because that day belongs to the year 2018. The weekofyear() functions returns 1 for the input '2018-12-31', because that day belongs to the first week of 2019. Both functions provide sensible results on their own, but when combined, the result is wrong, because '2018-12-31' does not belong to week 1 of 2018. I suggest adding a new function yearofweek() and adding a warning to the documentation of the weekofyear() function about this problem. > A day may belong to a different year than the week it is a part of > -- > > Key: HIVE-21117 > URL: https://issues.apache.org/jira/browse/HIVE-21117 > Project: Hive > Issue Type: New Feature >Affects Versions: 2.3.4 >Reporter: Zoltan Ivanfi >Priority: Major > > When using the year() and weekofyear() functions in a query, their result is > 2018 and 1 (respectively) for the day '2018-12-31'. > The year() function returns 2018 for the input '2018-12-31', because that day > belongs to the year 2018. > The weekofyear() functions returns 1 for the input '2018-12-31', because that > day belongs to the first week of 2019. > Both functions provide sensible results on their own, but when combined, the > result is wrong, because '2018-12-31' does not belong to week 1 of 2018. > I suggest adding a new function yearofweek() that would return 2019 for the > input '2018-12-31' and adding a warning to the documentation of the > weekofyear() function about this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20241) Support partitioning spec in CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-20241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740443#comment-16740443
]
Jesus Camacho Rodriguez commented on HIVE-20241:
Thanks [~b.maidics], I have updated the documentation.
> Support partitioning spec in CTAS statements
>
>
> Key: HIVE-20241
> URL: https://issues.apache.org/jira/browse/HIVE-20241
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: TODOC3.2
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-20241.01.patch, HIVE-20241.01.patch,
> HIVE-20241.01.patch, HIVE-20241.02.patch, HIVE-20241.03.patch,
> HIVE-20241.patch
>
>
> Currently, for partitioned tables we will declare the table and insert the
> data in different operations. This issue is to extend CTAS statement to
> support specifying partition columns.
> For instance:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740423#comment-16740423
]
Hive QA commented on HIVE-17084:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15591/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common ql accumulo-handler contrib hbase-handler .
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15591/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Turn on hive.stats.fetch.column.stats configuration flag
>
>
> Key: HIVE-17084
> URL: https://issues.apache.org/jira/browse/HIVE-17084
> Project: Hive
> Issue Type: Task
> Components: Statistics
>Reporter: Vineet Garg
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch,
> HIVE-17
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740411#comment-16740411
]
Marta Kuczora commented on HIVE-21095:
--
+1
Thanks a lot [~klcopp] for the patch.
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[
https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740415#comment-16740415
]
Hive QA commented on HIVE-17084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954581/HIVE-17084.26.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 400 failed/errored test(s), 15712 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=267)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=275)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_11]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join9] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats2]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binarysortable_1]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_SortUnionTransposeRule]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join1]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[column_pruner_multiple_children]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_quoting]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropagateForSubQuery]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog2] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer11]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer13]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer8]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer9]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_1]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cross_product_check_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_uses_database_location]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.test
[jira] [Commented] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740397#comment-16740397 ] Denys Kuzmenko commented on HIVE-21116: --- [~vihangk1], [~pvary] could you please review. Thank you! > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21116: -- Attachment: HIVE-21116.1.patch > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-21116: - Assignee: Denys Kuzmenko > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21001) Upgrade to calcite-1.18
[ https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21001: Attachment: HIVE-21001.06.patch > Upgrade to calcite-1.18 > --- > > Key: HIVE-21001 > URL: https://issues.apache.org/jira/browse/HIVE-21001 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21001.01.patch, HIVE-21001.01.patch, > HIVE-21001.02.patch, HIVE-21001.03.patch, HIVE-21001.04.patch, > HIVE-21001.05.patch, HIVE-21001.06.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17084) Turn on hive.stats.fetch.column.stats configuration flag
[ https://issues.apache.org/jira/browse/HIVE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-17084: Attachment: HIVE-17084.26.patch > Turn on hive.stats.fetch.column.stats configuration flag > > > Key: HIVE-17084 > URL: https://issues.apache.org/jira/browse/HIVE-17084 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-17084.08.patch, HIVE-17084.09.patch, > HIVE-17084.1.patch, HIVE-17084.10.patch, HIVE-17084.11.patch, > HIVE-17084.12.patch, HIVE-17084.13.patch, HIVE-17084.14.patch, > HIVE-17084.15.patch, HIVE-17084.16.patch, HIVE-17084.17.patch, > HIVE-17084.18.patch, HIVE-17084.19.patch, HIVE-17084.20.patch, > HIVE-17084.21.patch, HIVE-17084.22.patch, HIVE-17084.23.patch, > HIVE-17084.24.patch, HIVE-17084.25.patch, HIVE-17084.26.patch, > HIVE-170884.4.patch, HIVE-170884.5.patch, HIVE-170884.7.patch > > > This flag is off by default and could result in bad plans due to missing > column statistics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740339#comment-16740339
]
Jaume M commented on HIVE-21052:
The only problem I see with doing it at {{TxnHandler.enqueueLockWithRetry}} is
that at that point we don't know how many writeIds we have and we have to
assert that we only have allocated one writeId. Should this check be done at
allocateWriteIds and add a field isDynamicPartitioning to
AllocateTableWriteIdsRequest ?
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21114) Create read-only transactions
[
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko reassigned HIVE-21114:
Assignee: Igor Kryvenko
> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a
> read and mark the txn accordingly.
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt,
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this
> requires figuring out how to determine if a query is a SELECT. By the time
> {{Driver.openTransaction();}} is called, we have already parsed the query so
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to
> indicate that a txn is read-only and then not allow any statements that can
> make modifications in this txn. This should be a different jira.
> cc [~ikryvenko]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21063) Support statistics in cachedStore for transactional table
[
https://issues.apache.org/jira/browse/HIVE-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740280#comment-16740280
]
Hive QA commented on HIVE-21063:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954553/HIVE-21063.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15717 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testUpdatePartitionStat_doesNotUpdateStats
(batchId=223)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testUpdatePartitionStat_doesNotUpdateStats
(batchId=224)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testUpdatePartitionStat_doesNotUpdateStats
(batchId=227)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testUpdatePartitionStat_doesNotUpdateStats
(batchId=230)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testUpdatePartitionStat_doesNotUpdateStats
(batchId=221)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testUpdatePartitionStat_doesNotUpdateStats
(batchId=219)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testUpdatePartitionStat_doesNotUpdateStats
(batchId=229)
org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1
(batchId=259)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=261)
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15590/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954553 - PreCommit-HIVE-Build
> Support statistics in cachedStore for transactional table
> -
>
> Key: HIVE-21063
> URL: https://issues.apache.org/jira/browse/HIVE-21063
> Project: Hive
> Issue Type: Task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21063.01.patch
>
>
> Currently statistics for transactional table is not stored in cached store
> for consistency issues. Need to add validation for valid write ids and
> generation of aggregate stats based on valid partitions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20504) Give simple MJ bigger priority than bucketized ones
[ https://issues.apache.org/jira/browse/HIVE-20504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-20504: Resolution: Won't Fix Status: Resolved (was: Patch Available) this was addressed a bit differently; IIRC there was a bug in selecting BMJ when it was not entirely applicable... > Give simple MJ bigger priority than bucketized ones > --- > > Key: HIVE-20504 > URL: https://issues.apache.org/jira/browse/HIVE-20504 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20504.01.patch, HIVE-20504.01.patch, > HIVE-20504.01wip01.patch, HIVE-20504.01wip01.patch > > > from the code it seems "standard" mapjoin is one of the last one tried; in > case the table estimated to be bucketed in to 2 - but it's small ; Hive willl > do a bucketmapjoin or dphj...even thru a simple mapjoin could have been an > alternative... > https://github.com/apache/hive/blob/154ca3e3b5eb78cd49a4b3650c750ca731fba7da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L157 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21063) Support statistics in cachedStore for transactional table
[
https://issues.apache.org/jira/browse/HIVE-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740243#comment-16740243
]
Hive QA commented on HIVE-21063:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2309 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 15 unchanged - 1 fixed
= 16 total (was 16) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 15
unchanged - 5 fixed = 19 total (was 20) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 187 unchanged - 1 fixed = 187 total (was 188) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} server-extensions in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15590/dev-support/hive-personality.sh
|
| git revision | master / f713140 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15590/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15590/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore/metastore-server ql
hcatalog/server-extensions itests
[jira] [Updated] (HIVE-21063) Support statistics in cachedStore for transactional table
[ https://issues.apache.org/jira/browse/HIVE-21063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21063: --- Status: Patch Available (was: Open) > Support statistics in cachedStore for transactional table > - > > Key: HIVE-21063 > URL: https://issues.apache.org/jira/browse/HIVE-21063 > Project: Hive > Issue Type: Task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Attachments: HIVE-21063.01.patch > > > Currently statistics for transactional table is not stored in cached store > for consistency issues. Need to add validation for valid write ids and > generation of aggregate stats based on valid partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
