[jira] [Commented] (HIVE-18344) Remove LinkedList from SharedWorkOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-18344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396580#comment-16396580
]
Hive QA commented on HIVE-18344:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903734/HIVE-18344.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9614/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9614/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-13 05:47:43.123
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9614/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-13 05:47:43.126
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 12041d3 HIVE-17990 Add Thrift and DB storage for Schema Registry
objects (Alan Gates, reviewed by Thejas Nair)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 12041d3 HIVE-17990 Add Thrift and DB storage for Schema Registry
objects (Alan Gates, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-13 05:47:46.384
+ rm -rf ../yetus_PreCommit-HIVE-Build-9614
+ mkdir ../yetus_PreCommit-HIVE-Build-9614
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9614
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9614/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:
does not exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:392
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java' with
conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java:392
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903734 - PreCommit-HIVE-Build
> Remove LinkedList from SharedWorkOptimizer.java
> ---
>
> Key: HIVE-18344
> URL: https://issues.apache.org/jira/browse/HIVE-18344
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18344.1.patch
>
>
> Prefer {{ArrayList}} over {{LinkedList}} especially in this class because the
> initial size of the collection is known.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396578#comment-16396578
]
Hive QA commented on HIVE-18343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914189/HIVE-18343.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13002 tests
executed
*Failed tests:*
{noformat}
TestDFSErrorHandling - did not produce a TEST-*.xml file (likely timed out)
(batchId=237)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=92)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[ https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18910: -- Attachment: HIVE-18910.5.patch > Migrate to Murmur hash for shuffle and bucketing > > > Key: HIVE-18910 > URL: https://issues.apache.org/jira/browse/HIVE-18910 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18910.1.patch, HIVE-18910.2.patch, > HIVE-18910.3.patch, HIVE-18910.4.patch, HIVE-18910.5.patch > > > Hive uses JAVA hash which is not as good as murmur for better distribution > and efficiency in bucketing a table. > Migrate to murmur hash but still keep backward compatibility for existing > users so that they dont have to reload the existing tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18927) Hive "insert overwrite" doesn't replace the destination files if no partition in metastore for the files
[
https://issues.apache.org/jira/browse/HIVE-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangzhihao updated HIVE-18927:
--
Description:
[This
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
describes a way to reproduce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1
# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x 3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r-- 3 hdfs supergroup 8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has
no partition for the files, the {{oldPath}} is null and thus the files get no
chance to be cleaned. We should also clean {{destf}} in method
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
to fix the issue.
Also note, dynamic partition insertion doesn't have this issue. Only static
partition insertion has the issue.
was:
[This
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
describes a way to reproduce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1
# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x 3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r-- 3 hdfs supergroup 8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has
no partition for the files, the {{oldPath}} is null and thus the files get no
chance to be cleaned. We should also clean {{destf}} in method
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
to fix the issue.
> Hive "insert overwrite" doesn't replace the destination files if no partition
> in metastore for the files
>
>
> Key: HIVE-18927
> URL: https://issues.apache.org/jira/browse/HIVE-18927
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: wangzhihao
>Priority: Major
>
> [This
> post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
> describes a way to reproduce this issue:
> {noformat}
> # Add some files into file system but no partition in metastore to track it.
> hdfs dfs -put test.txt test/p=p1
> # Insert overwrite the partition(p = p1)
> DROP TABLE IF EXISTS partition_test;
> CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
> INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
> # verify the text.txt is not removed.
> hdfs dfs -ls test/p=p1
> Found 2 items
> -rwxr-xr-x 3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
> -rw-r--r-- 3 hdfs supergroup 8 2015-05-05 00:10 test/p=p1/test.txt
> {noformat}
> The reason is that
> [Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
> will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has
> no partition for the files, the {{oldPath}} is null and thus the files get no
> chance to be cleaned. We should also clean {{destf}} in method
> [Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
> to fix the issue.
> Also note, dynamic partition insertion doesn't have this issue. Only static
> partition insertion has the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396543#comment-16396543
]
Hive QA commented on HIVE-18343:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 2 unchanged - 43 fixed
= 3 total (was 45) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9613/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9613/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9613/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9613/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch, HIVE-18343.4.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18768) Use Datanucleus to serialize notification updates
[ https://issues.apache.org/jira/browse/HIVE-18768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396533#comment-16396533 ] Alexander Kolbasov commented on HIVE-18768: --- Th3 same thing runs fine in branch-2. > Use Datanucleus to serialize notification updates > - > > Key: HIVE-18768 > URL: https://issues.apache.org/jira/browse/HIVE-18768 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.2, 3.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-18768.01.patch, HIVE-18768.02.patch > > > HIVE-16886 added code to serialize notification updates using LOCK FOR > UPDATE. It turns out that there is a simpler way - see HIVE-18526. The goal > of this JIRA is to use the approach from HIVE-18526 - Datanucleus based > solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18768) Use Datanucleus to serialize notification updates
[
https://issues.apache.org/jira/browse/HIVE-18768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396523#comment-16396523
]
Alexander Kolbasov commented on HIVE-18768:
---
Interesting - all DbNotificationListener tests in branch 3 are now failing here:
{code}
2018-03-12T20:52:22,775 ERROR [main] metastore.ObjectStore: couldnot get lock
for update
java.lang.NullPointerException: null
at
org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:)
[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at
org.apache.hive.hcatalog.listener.DummyRawStoreFailEvent.addNotificationEvent(DummyRawStoreFailEvent.java:819)
[test-classes/:?]
{code}
This used to work before - I am not sure what actually changed in the test
setup. [~vihangk1] any idea?
> Use Datanucleus to serialize notification updates
> -
>
> Key: HIVE-18768
> URL: https://issues.apache.org/jira/browse/HIVE-18768
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.2, 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-18768.01.patch, HIVE-18768.02.patch
>
>
> HIVE-16886 added code to serialize notification updates using LOCK FOR
> UPDATE. It turns out that there is a simpler way - see HIVE-18526. The goal
> of this JIRA is to use the approach from HIVE-18526 - Datanucleus based
> solution.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18885) Cascaded alter table + notifications = disaster
[
https://issues.apache.org/jira/browse/HIVE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396494#comment-16396494
]
Alexander Kolbasov commented on HIVE-18885:
---
Note that th issue may happen on Hive-2 branch even for non-transactional
listeners because they can easily be called in transactional context for nested
transactions.
> Cascaded alter table + notifications = disaster
> ---
>
> Key: HIVE-18885
> URL: https://issues.apache.org/jira/browse/HIVE-18885
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 2.3.2
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> You can see the problem from looking at the code, but it actually created
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
> msdb.openTransaction()
> ...
> List parts = msdb.getPartitions(dbname, name, -1);
> for (Partition part : parts) {
> List oldCols = part.getSd().getCols();
> part.getSd().setCols(newt.getSd().getCols());
> String oldPartName =
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
> updatePartColumnStatsForAlterColumns(msdb, part, oldPartName,
> part.getValues(), oldCols, part);
> msdb.alterPartition(dbname, name, part.getValues(), part);
> }
> {code}
> So it walks all partitions (and this may be huge list) and does some
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition,
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is
> happening no other write DDL can proceed. This can sometimes cause DB lock
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[
https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396483#comment-16396483
]
Hive QA commented on HIVE-18933:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} llap-server: The patch generated 0 new + 33
unchanged - 3 fixed = 33 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9612/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9612/yetus/patch-asflicense-problems.txt
|
| modules | C: common llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9612/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> disable ORC codec pool for now; remove clone
>
>
> Key: HIVE-18933
> URL: https://issues.apache.org/jira/browse/HIVE-18933
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18933.patch
>
>
> See ORC-310.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.03.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: In Progress) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch, > HIVE-18908.03.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: In Progress (was: Patch Available) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[
https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396470#comment-16396470
]
Hive QA commented on HIVE-16992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914186/HIVE-16992.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 13409 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396451#comment-16396451
]
Alexander Kolbasov commented on HIVE-18940:
---
[~LinaAtAustin] [~kkalyan] [~spena] FYI.
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-11111) Insert on skewed table with STORED AS DIRECTORY is broken
[
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396445#comment-16396445
]
NeilWang commented on HIVE-1:
-
I also have this problem where select data from the skew table.Can you offer me
some advice?
> Insert on skewed table with STORED AS DIRECTORY is broken
> -
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Damien Carol
>Priority: Major
>
> Doing these queries fails:
> {code:sql}
> RESET;
> DROP TABLE IF EXISTS testskew;
> CREATE TABLE IF NOT EXISTS testskew (key int, value STRING)
> SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES
> STORED AS ORC;
> insert into testskew VALUES
> (1, 'one'),
> (1, 'one'),
> (1, 'one'),
> (1, 'one'),
> (1, 'one'),
> (1, 'one'),
> (2, 'two'),
> (3, 'three'),
> (5, 'five'),
> (5, 'five'),
> (5, 'five'),
> (5, 'five'),
> (5, 'five'),
> (6, 'six'),
> (6, 'six'),
> (6, 'six'),
> (6, 'six'),
> (6, 'six'),
> (6, 'six');
> {code}
> Stacktrace:
> {noformat}
> INFO : Session is already open
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1434957292922_0059)
> INFO : Map 1: 0/1
> INFO : Map 1: 0(+1)/1
> INFO : Map 1: 1/1
> INFO : Loading data to table test.testskew from
> hdfs://nc-h07/user/hive/warehouse/test.db/testskew/.hive-staging_hive_2015-06-25_17-29-34_385_4424227988595852796-14/-ext-1
> ERROR : Failed with exception checkPaths:
> hdfs://nc-h07/user/hive/warehouse/test.db/testskew/.hive-staging_hive_2015-06-25_17-29-34_385_4424227988595852796-14/-ext-1
> has nested directory
> hdfs://nc-h07/user/hive/warehouse/test.db/testskew/.hive-staging_hive_2015-06-25_17-29-34_385_4424227988595852796-14/-ext-1/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
> org.apache.hadoop.hive.ql.metadata.HiveException: checkPaths:
> hdfs://nc-h07/user/hive/warehouse/test.db/testskew/.hive-staging_hive_2015-06-25_17-29-34_385_4424227988595852796-14/-ext-1
> has nested directory
> hdfs://nc-h07/user/hive/warehouse/test.db/testskew/.hive-staging_hive_2015-06-25_17-29-34_385_4424227988595852796-14/-ext-1/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
> at org.apache.hadoop.hive.ql.metadata.Hive.checkPaths(Hive.java:2466)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2701)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1645)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:297)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18533: Attachment: HIVE-18533.2.patch > Add option to use InProcessLauncher to submit spark jobs > > > Key: HIVE-18533 > URL: https://issues.apache.org/jira/browse/HIVE-18533 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch > > > See discussion in HIVE-16484 for details. > I think this will help with reducing the amount of time it takes to open a > HoS session + debuggability (no need launch a separate process to run a Spark > app). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone
[
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18925:
---
Assignee: Piotr Findeisen
> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
> Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Assignee: Piotr Findeisen
>Priority: Major
> Labels: pull-request-available
>
> Hive Server2 doesn't work if started with
> {{-Duser.timezone=America/Bahia_Banderas}}
>
> Steps to reproduce
> # use [https://github.com/big-data-europe/docker-hive]
> # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to
> {{hive-server}} docker container environment configuration
> # {{docker-compose up}}
> #
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think
> it's initialization of
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone
[
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396409#comment-16396409
]
Sergey Shelukhin commented on HIVE-18925:
-
[~findepi] can you attach a patch file named e.g. HIVE-18925.patch and hit
submit patch? HiveQA picks up patches from JIRA rather than using pull requests
> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
> Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
> Labels: pull-request-available
>
> Hive Server2 doesn't work if started with
> {{-Duser.timezone=America/Bahia_Banderas}}
>
> Steps to reproduce
> # use [https://github.com/big-data-europe/docker-hive]
> # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to
> {{hive-server}} docker container environment configuration
> # {{docker-compose up}}
> #
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with
> org.joda.time.IllegalInstantException: Illegal instant due to time zone
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think
> it's initialization of
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[
https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396407#comment-16396407
]
Hive QA commented on HIVE-16992:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} llap-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} llap-common: The patch generated 7 new + 13 unchanged -
0 fixed = 20 total (was 13) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 18 new + 244
unchanged - 6 fixed = 262 total (was 250) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 3 new + 165 unchanged - 1
fixed = 168 total (was 166) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9611/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus/patch-mvninstall-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus/diff-checkstyle-llap-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus/diff-checkstyle-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api common llap-common llap-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9611/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: monitoring and better default lambda for LRFU policy
> --
>
> Key: HIVE-16992
> URL: https://issues.apache.org/jira/browse/HIVE-16992
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-16992.patch
>
>
> LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of
> buffers tracked during a typical workload,
[jira] [Commented] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396405#comment-16396405 ] Sergey Shelukhin commented on HIVE-18919: - [~ashutoshc] can you take a look? small patch > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18919.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18919: Status: Patch Available (was: Open) > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18919.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18919: Attachment: HIVE-18919.patch > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18919.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396401#comment-16396401
]
Vihang Karajgaonkar commented on HIVE-17580:
patch merged to standalone-metastore branch. Thanks [~alangates] for the review
and [~owen.omalley] for the guidance.
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch,
> HIVE-17580.05-standalone-metastore.patch,
> HIVE-17580.06-standalone-metastore.patch,
> HIVE-17580.07-standalone-metastore.patch,
> HIVE-17580.08-standalone-metastore.patch,
> HIVE-17580.09-standalone-metastore.patch,
> HIVE-17580.092-standalone-metastore.patch,
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17580:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch,
> HIVE-17580.05-standalone-metastore.patch,
> HIVE-17580.06-standalone-metastore.patch,
> HIVE-17580.07-standalone-metastore.patch,
> HIVE-17580.08-standalone-metastore.patch,
> HIVE-17580.09-standalone-metastore.patch,
> HIVE-17580.092-standalone-metastore.patch,
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18934) LLAP: consider a simpler cache policy (SLRU?)
[ https://issues.apache.org/jira/browse/HIVE-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18934: Description: LRFU as it is is not so easy to tune and its behavior is not intuitive from users perspective, esp. w.r.t. full scan resistance. We can consider simpler policies with more intuitive behavior, like segmented LRU. was: LRFU as it is is not so easy to tune and it's behavior is not intuitive from users perspective, esp. w.r.t. full scan resistance. We can consider simpler policies with more intuitive behavior, like segmented LRU. > LLAP: consider a simpler cache policy (SLRU?) > - > > Key: HIVE-18934 > URL: https://issues.apache.org/jira/browse/HIVE-18934 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > LRFU as it is is not so easy to tune and its behavior is not intuitive from > users perspective, esp. w.r.t. full scan resistance. > We can consider simpler policies with more intuitive behavior, like segmented > LRU. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18934) LLAP: consider a simpler cache policy (SLRU?)
[ https://issues.apache.org/jira/browse/HIVE-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18934: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: consider a simpler cache policy (SLRU?) > - > > Key: HIVE-18934 > URL: https://issues.apache.org/jira/browse/HIVE-18934 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > LRFU as it is is not so easy to tune and it's behavior is not intuitive from > users perspective, esp. w.r.t. full scan resistance. > We can consider simpler policies with more intuitive behavior, like segmented > LRU. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16992: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: monitoring and better default lambda for LRFU policy > -- > > Key: HIVE-16992 > URL: https://issues.apache.org/jira/browse/HIVE-16992 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-16992.patch > > > LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of > buffers tracked during a typical workload, but the heap size is around 700. > We should see if making it closer to LFU (by tweaking the lambda) will > improve hit rate with small queries infrequently interleaved with large > scans; and whether it will have negative effects due to perf overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18937) LLAP: management API to dump cache on one node
[ https://issues.apache.org/jira/browse/HIVE-18937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18937: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: management API to dump cache on one node > -- > > Key: HIVE-18937 > URL: https://issues.apache.org/jira/browse/HIVE-18937 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18936) LLAP: improve data tags for cache view to not be based on paths
[ https://issues.apache.org/jira/browse/HIVE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18936: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: improve data tags for cache view to not be based on paths > --- > > Key: HIVE-18936 > URL: https://issues.apache.org/jira/browse/HIVE-18936 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > Due to Hive splits limitations (the only thing they have is file paths) the > database/table/partition tags for cache entries (see HIVE-16992) are > guesswork derived from file paths. > We should improve them; in some cases where split code is in Hive (e.g. > OrcSplit) we can add information directly to the split. > For others, we can consider passing information as part of JobConf or in some > other way. Hive execution already has path to partition mapping, so we may be > able to utilize that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18936) LLAP: improve data tags for cache view
[ https://issues.apache.org/jira/browse/HIVE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18936: Description: Due to Hive splits limitations (the only thing they have is file paths) the database/table/partition tags for cache entries (see HIVE-16992) are guesswork derived from file paths. We should improve them; in some cases where split code is in Hive (e.g. OrcSplit) we can add information directly to the split. For others, we can consider passing information as part of JobConf or in some other way. Hive execution already has path to partition mapping, so we may be able to utilize that. was: Due to splits limitations the database/table/partition tags for cache entries (see HIVE-16992) are guesswork derived from file paths. We should improve them; in some cases where split code is in Hive (e.g. OrcSplit) we can add information directly to the split. For others, we can consider passing information as part of JobConf or in some other way. > LLAP: improve data tags for cache view > -- > > Key: HIVE-18936 > URL: https://issues.apache.org/jira/browse/HIVE-18936 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Due to Hive splits limitations (the only thing they have is file paths) the > database/table/partition tags for cache entries (see HIVE-16992) are > guesswork derived from file paths. > We should improve them; in some cases where split code is in Hive (e.g. > OrcSplit) we can add information directly to the split. > For others, we can consider passing information as part of JobConf or in some > other way. Hive execution already has path to partition mapping, so we may be > able to utilize that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18936) LLAP: improve data tags for cache view to not be based on paths
[ https://issues.apache.org/jira/browse/HIVE-18936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18936: Summary: LLAP: improve data tags for cache view to not be based on paths (was: LLAP: improve data tags for cache view) > LLAP: improve data tags for cache view to not be based on paths > --- > > Key: HIVE-18936 > URL: https://issues.apache.org/jira/browse/HIVE-18936 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Due to Hive splits limitations (the only thing they have is file paths) the > database/table/partition tags for cache entries (see HIVE-16992) are > guesswork derived from file paths. > We should improve them; in some cases where split code is in Hive (e.g. > OrcSplit) we can add information directly to the split. > For others, we can consider passing information as part of JobConf or in some > other way. Hive execution already has path to partition mapping, so we may be > able to utilize that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18935) LLAP: add jmx view of what's in cache
[ https://issues.apache.org/jira/browse/HIVE-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18935: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: add jmx view of what's in cache > - > > Key: HIVE-18935 > URL: https://issues.apache.org/jira/browse/HIVE-18935 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > Based on HIVE-16992 work -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18939) LLAP cache improvements (local)
[ https://issues.apache.org/jira/browse/HIVE-18939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18939: --- > LLAP cache improvements (local) > --- > > Key: HIVE-18939 > URL: https://issues.apache.org/jira/browse/HIVE-18939 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18938) LLAP: cache priorities (soft pin, exclusions) configuration
[ https://issues.apache.org/jira/browse/HIVE-18938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18938: Issue Type: Sub-task (was: Bug) Parent: HIVE-18939 > LLAP: cache priorities (soft pin, exclusions) configuration > --- > > Key: HIVE-18938 > URL: https://issues.apache.org/jira/browse/HIVE-18938 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > > We want persistent configuration and not an ad-hoc pin API because the latter > is both unreliable (e.g. node might go down and restart, so someone has to > keep track of global state anyway - might as well have it done explicitly), > and also dangerous because errant pin calls can fill cache with garbage and > make it unusable. > So, the idea is that we can allow to > 1) Give some tables or partitions (tags, see HIVE-16992) a priority boost at > cache time. > 2) Exclude some tables or partitions from being cached. > Ideally we also want this changeable without LLAP restart (and managed by > HS2). So the choice here is either make this a config setting, like many > Hadoop settings that have user/group/etc lists, and implement > ZK-based/HS2-based configuration changes without restart, that we want to do > anyway at some point; > or make this a separate thing, probably managed and stored as policies in > metastore (kind of like workload management) and update via ZK. > It's possible to go with config settings that do require restart for the > first cut. > Separately, we can add a workload management policy that would disable cache > for certain query pools (i.e. users/groups/apps). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396381#comment-16396381
]
Hive QA commented on HIVE-18034:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914174/HIVE-18034.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13409 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Assigned] (HIVE-18935) LLAP: add jmx view of what's in cache
[ https://issues.apache.org/jira/browse/HIVE-18935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18935: --- > LLAP: add jmx view of what's in cache > - > > Key: HIVE-18935 > URL: https://issues.apache.org/jira/browse/HIVE-18935 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > Based on HIVE-16992 work -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18344) Remove LinkedList from SharedWorkOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-18344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396353#comment-16396353
]
Sahil Takiar commented on HIVE-18344:
-
Think this needs to be rebased. Patch doesn't apply cleanly.
> Remove LinkedList from SharedWorkOptimizer.java
> ---
>
> Key: HIVE-18344
> URL: https://issues.apache.org/jira/browse/HIVE-18344
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18344.1.patch
>
>
> Prefer {{ArrayList}} over {{LinkedList}} especially in this class because the
> initial size of the collection is known.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18344) Remove LinkedList from SharedWorkOptimizer.java
[
https://issues.apache.org/jira/browse/HIVE-18344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396343#comment-16396343
]
BELUGA BEHR commented on HIVE-18344:
[~stakiar] :)
> Remove LinkedList from SharedWorkOptimizer.java
> ---
>
> Key: HIVE-18344
> URL: https://issues.apache.org/jira/browse/HIVE-18344
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18344.1.patch
>
>
> Prefer {{ArrayList}} over {{LinkedList}} especially in this class because the
> initial size of the collection is known.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18919) remove separate keytab setting for ZK in LLAP
[ https://issues.apache.org/jira/browse/HIVE-18919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18919: --- Assignee: Sergey Shelukhin > remove separate keytab setting for ZK in LLAP > - > > Key: HIVE-18919 > URL: https://issues.apache.org/jira/browse/HIVE-18919 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18907: -- Component/s: ORC > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug > Components: ORC, Transactions >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396324#comment-16396324
]
Hive QA commented on HIVE-18034:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
53s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 3 new + 50 unchanged - 19
fixed = 53 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} spark-client: The patch generated 1 new + 30 unchanged
- 3 fixed = 31 total (was 33) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9610/dev-support/hive-personality.sh
|
| git revision | master / 12041d3 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9610/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9610/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9610/yetus/diff-checkstyle-spark-client.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9610/yetus/patch-asflicense-problems.txt
|
| modules | C: ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9610/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18034.1.patch, HIVE-18034.2.patch,
> HIVE-18034.3.patch, HIVE-18034.4.patch, HIVE-18034.6.patch, HIVE-18034.7.patch
>
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18343:
---
Status: Open (was: Patch Available)
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch, HIVE-18343.4.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18343:
---
Status: Patch Available (was: Open)
Remove a bit more superfluous code
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch, HIVE-18343.4.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18343:
---
Attachment: HIVE-18343.4.patch
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch, HIVE-18343.4.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18933: Status: Patch Available (was: Open) > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396313#comment-16396313 ] Sergey Shelukhin commented on HIVE-18933: - [~prasanth_j] can you take a look? cc [~owen.omalley] I'm not sure why these clone calls were added. > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18933: Attachment: HIVE-18933.patch > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18933.patch > > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396306#comment-16396306
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914170/HIVE-18908.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9609/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9609/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9609/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-3.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/util/ToolRunner.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/CancellationException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/conf/Configured.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/io/NullWritable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/io/Text.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapred/JobClient.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapred/JobConf.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapreduce/Job.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapreduce/JobID.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.0.0-beta1/hadoop-mapreduce-client-core-3.0.0-beta1.jar(org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/security/token/Token.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/util/Tool.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar(org/apache/thrift/TException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.0.0-beta1/hadoop-common-3.0.0-beta1.jar(org/apache/hadoop/conf/Configurable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Callable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/InterruptedException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Boolean.class)]]
[jira] [Assigned] (HIVE-18933) disable ORC codec pool for now; remove clone
[ https://issues.apache.org/jira/browse/HIVE-18933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18933: --- > disable ORC codec pool for now; remove clone > > > Key: HIVE-18933 > URL: https://issues.apache.org/jira/browse/HIVE-18933 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > See ORC-310. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396293#comment-16396293
]
Hive QA commented on HIVE-18693:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914161/HIVE-18693.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-13 00:23:36.715
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9608/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-13 00:23:36.718
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
d717d38..12041d3 master -> origin/master
+ git reset --hard HEAD
HEAD is now at d717d38 HIVE-18907: Create utility to fix acid key index issue
from HIVE-18817 (Jason Dere, reviewed by Prasanth Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 12041d3 HIVE-17990 Add Thrift and DB storage for Schema Registry
objects (Alan Gates, reviewed by Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-13 00:23:39.689
+ rm -rf ../yetus_PreCommit-HIVE-Build-9608
+ mkdir ../yetus_PreCommit-HIVE-Build-9608
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9608
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9608/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp:15587
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp' with
conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:454: trailing whitespace.
tmpMap.put(_Fields.IS_TRANSACTIONAL, new
org.apache.thrift.meta_data.FieldMetaData("isTransactional",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
error: patch failed:
standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp:15587
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp' with
conflicts.
U standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
warning: 1 line adds whitespace errors.
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12914161 - PreCommit-HIVE-Build
> Snapshot Isolation does not work for Micromanaged table when a insert
> transaction is aborted
>
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch,
> HIVE-18693.03.patch, HIVE-18693.04.patch, HIVE-18693.05.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results.
> But this test works fine with full
[jira] [Updated] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18907: -- Component/s: Transactions > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396291#comment-16396291
]
Hive QA commented on HIVE-18778:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914114/HIVE-18788_SparkPerf.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 198 failed/errored test(s), 13756 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
(batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_table_failure5]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_collect_set_unsupported]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_array_contains_wrong2]
(batchId=95)
[jira] [Commented] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396276#comment-16396276 ] Sergey Shelukhin commented on HIVE-16992: - Also this patch needs some cluster testing, I did a bunch of a debug version before refactoring though. > LLAP: monitoring and better default lambda for LRFU policy > -- > > Key: HIVE-16992 > URL: https://issues.apache.org/jira/browse/HIVE-16992 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-16992.patch > > > LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of > buffers tracked during a typical workload, but the heap size is around 700. > We should see if making it closer to LFU (by tweaking the lambda) will > improve hit rate with small queries infrequently interleaved with large > scans; and whether it will have negative effects due to perf overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16992: Status: Patch Available (was: Open) I will file some follow-up jiras from this. 1) Better mapping for entities in moitoring, not just based on the split path. 2) Easier to tune/simpler cache policy :) > LLAP: monitoring and better default lambda for LRFU policy > -- > > Key: HIVE-16992 > URL: https://issues.apache.org/jira/browse/HIVE-16992 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-16992.patch > > > LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of > buffers tracked during a typical workload, but the heap size is around 700. > We should see if making it closer to LFU (by tweaking the lambda) will > improve hit rate with small queries infrequently interleaved with large > scans; and whether it will have negative effects due to perf overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16992: Attachment: HIVE-16992.patch > LLAP: monitoring and better default lambda for LRFU policy > -- > > Key: HIVE-16992 > URL: https://issues.apache.org/jira/browse/HIVE-16992 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-16992.patch > > > LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of > buffers tracked during a typical workload, but the heap size is around 700. > We should see if making it closer to LFU (by tweaking the lambda) will > improve hit rate with small queries infrequently interleaved with large > scans; and whether it will have negative effects due to perf overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16992) LLAP: monitoring and better default lambda for LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16992: Summary: LLAP: monitoring and better default lambda for LRFU policy (was: LLAP: better default lambda for LRFU policy) > LLAP: monitoring and better default lambda for LRFU policy > -- > > Key: HIVE-16992 > URL: https://issues.apache.org/jira/browse/HIVE-16992 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of > buffers tracked during a typical workload, but the heap size is around 700. > We should see if making it closer to LFU (by tweaking the lambda) will > improve hit rate with small queries infrequently interleaved with large > scans; and whether it will have negative effects due to perf overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17990: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Patch 3 committed. Thank you Thejas for the review. > Add Thrift and DB storage for Schema Registry objects > - > > Key: HIVE-17990 > URL: https://issues.apache.org/jira/browse/HIVE-17990 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: Adding-Schema-Registry-to-Metastore.pdf, > HIVE-17990.2.patch, HIVE-17990.3.patch, HIVE-17990.patch > > > This JIRA tracks changes to Thrift, RawStore, and DB scripts to support > objects in the Schema Registry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396254#comment-16396254
]
Sahil Takiar commented on HIVE-18916:
-
hmm yeah I'll have to dig out the stack-trace. I can't remember exactly how I
triggered this, so let me double check
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL
[ https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18675: -- Component/s: Metastore > make HIVE_LOCKS.HL_TXNID NOT NULL > - > > Key: HIVE-18675 > URL: https://issues.apache.org/jira/browse/HIVE-18675 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Kryvenko Igor >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18675.01.patch, HIVE-18675.02.patch > > > In Hive 3.0 all statements that may need locks run in a transaction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18675) make HIVE_LOCKS.HL_TXNID NOT NULL
[ https://issues.apache.org/jira/browse/HIVE-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18675: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master thanks Igor for the contribution > make HIVE_LOCKS.HL_TXNID NOT NULL > - > > Key: HIVE-18675 > URL: https://issues.apache.org/jira/browse/HIVE-18675 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Kryvenko Igor >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18675.01.patch, HIVE-18675.02.patch > > > In Hive 3.0 all statements that may need locks run in a transaction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396227#comment-16396227
]
Hive QA commented on HIVE-18778:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
39s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9606/dev-support/hive-personality.sh
|
| git revision | master / d717d38 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9606/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9606/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18433) Upgrade version of com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/HIVE-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396218#comment-16396218
]
Hive QA commented on HIVE-18433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914102/HIVE-18433.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12960 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18433) Upgrade version of com.fasterxml.jackson
[
https://issues.apache.org/jira/browse/HIVE-18433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396215#comment-16396215
]
Hive QA commented on HIVE-18433:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
12s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 52 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9605/dev-support/hive-personality.sh
|
| git revision | master / 1629ec0 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9605/yetus/patch-asflicense-problems.txt
|
| modules | C: common . druid-handler hcatalog/core hcatalog/server-extensions
hcatalog/webhcat/svr itests/hive-blobstore itests/qtest-druid ql spark-client
standalone-metastore testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9605/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade version of com.fasterxml.jackson
>
>
> Key: HIVE-18433
> URL: https://issues.apache.org/jira/browse/HIVE-18433
> Project: Hive
> Issue Type: Task
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18433.1.patch, HIVE-18433.2.patch,
> HIVE-18433.3.patch, HIVE-18433.4.patch, HIVE-18433.5.patch
>
>
> Let's upgrade to version 2.9.2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[ https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-18034: Attachment: HIVE-18034.7.patch > Improving logging with HoS executors spend lots of time in GC > - > > Key: HIVE-18034 > URL: https://issues.apache.org/jira/browse/HIVE-18034 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18034.1.patch, HIVE-18034.2.patch, > HIVE-18034.3.patch, HIVE-18034.4.patch, HIVE-18034.6.patch, HIVE-18034.7.patch > > > There are times when Spark will spend lots of time doing GC. The Spark > History UI shows a bunch of red flags when too much time is spent in GC. It > would be nice if those warnings are propagated to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18932) Better console logging for lifecycle of a Spark job
[ https://issues.apache.org/jira/browse/HIVE-18932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar reassigned HIVE-18932: --- > Better console logging for lifecycle of a Spark job > --- > > Key: HIVE-18932 > URL: https://issues.apache.org/jira/browse/HIVE-18932 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > There are a couple of improvements that can be made to the console logging > for the lifecycle of a Spark job. Like logs when the job is submitted, > queued, running, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18907: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396189#comment-16396189 ] Jason Dere commented on HIVE-18907: --- Committed to master > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Status: Patch Available (was: Open) Still have wrong results in vector_full_outer_join.q, fullouter_mapjoin_1_optimized.q, vector_leftsemi_mapjoin.q, vector_fullouter_mapjoin_1_optimized_passthru.q But give it a first spin anyway. > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: HIVE-18908.02.patch > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch, HIVE-18908.02.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18908) Add support for FULL OUTER JOIN to MapJoin
[ https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-18908: Attachment: (was: HIVE-18908.02.patch) > Add support for FULL OUTER JOIN to MapJoin > -- > > Key: HIVE-18908 > URL: https://issues.apache.org/jira/browse/HIVE-18908 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-18908.01.patch > > > Currently, we do not support FULL OUTER JOIN in MapJoin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14792:
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just pushed the addendum to those 3 branches.
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
> Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch,
> HIVE-14792.4.patch, HIVE-14792.5.patch, HIVE-14792.patch.addendum
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18899) Separate FetchWork required for each query that uses the results cache
[ https://issues.apache.org/jira/browse/HIVE-18899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18899: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master > Separate FetchWork required for each query that uses the results cache > -- > > Key: HIVE-18899 > URL: https://issues.apache.org/jira/browse/HIVE-18899 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Labels: performance > Fix For: 3.0.0 > > Attachments: HIVE-18899.1.patch, HIVE-18899.2.patch > > > [~gopalv] found issues when running lots of concurrent queries against HS2 > with the query cache. Looks like the FetchWork held by the results cache > cannot be shared between multiple queries because it contains a > ListSinkOperator that is used to hold the results of a fetch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[ https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Yeom updated HIVE-18693: -- Attachment: HIVE-18693.05.patch > Snapshot Isolation does not work for Micromanaged table when a insert > transaction is aborted > > > Key: HIVE-18693 > URL: https://issues.apache.org/jira/browse/HIVE-18693 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch, > HIVE-18693.03.patch, HIVE-18693.04.patch, HIVE-18693.05.patch > > > TestTxnCommands2#writeBetweenWorkerAndCleaner with minor > changes (changing delete command to insert command) fails on MM table. > Specifically the last SELECT commands returns wrong results. > But this test works fine with full ACID table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14792:
Attachment: HIVE-14792.patch.addendum
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
> Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch,
> HIVE-14792.4.patch, HIVE-14792.5.patch, HIVE-14792.patch.addendum
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395980#comment-16395980
]
Hive QA commented on HIVE-17990:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914100/HIVE-17990.3.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13405 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395896#comment-16395896
]
Sahil Takiar commented on HIVE-18343:
-
Can you create an RB for this? Its hard to review these types of checkstyle
exceptions without a good code-review tool.
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table
[
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395861#comment-16395861
]
BELUGA BEHR commented on HIVE-5795:
---
This is an oldie, but a goodie.
I was looking at this ticket and trying to decide if I should propose that we
drop this feature from Hive 3.0. I saw that there are some reports here of
there being issues... a suggestion that this should be an "experimental"
feature. For how long should that go on for?
I just ran some sample data threw this feature using Google's
[NGram|http://storage.googleapis.com/books/ngrams/books/datasetsv2.html] data.
I converted it to be comma separated (CSV) instead of tab separated (TSV)
before loading it into HDFS. I also added a header to each file in the Hive
table: {{ngram,year,match_count,volume_count}}.
{code:sql}
-- 2.2 GB / 1 File
CREATE EXTERNAL TABLE ngram1 (
ngram string,
year int,
match_count int,
volume_count int
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/admin/ngram1';
-- 2.2 GB / 1 File
CREATE EXTERNAL TABLE ngram2 (
ngram string,
year int,
match_count int,
volume_count int
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/admin/ngram2'
tblproperties ("skip.header.line.count"="1");
-- 5 Files / Each file has 1 header + 9 rows of data (10 total rows)
CREATE EXTERNAL TABLE ngram3 (
ngram string,
year int,
match_count int,
volume_count int
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/admin/ngram3'
tblproperties ("skip.header.line.count"="1");
{code}
{code:java|title=select count(1) from ngram1}
INFO : Compiling
command(queryId=hive_20180312132727_5a75ce57-53e5-4ab2-9b97-31f8b3502969):
select count(1) from ngram1
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20180312132727_5a75ce57-53e5-4ab2-9b97-31f8b3502969); Time
taken: 0.047 seconds
INFO : Executing
command(queryId=hive_20180312132727_5a75ce57-53e5-4ab2-9b97-31f8b3502969):
select count(1) from ngram1
INFO : Query ID = hive_20180312132727_5a75ce57-53e5-4ab2-9b97-31f8b3502969
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=
INFO : number of splits:9
INFO : Submitting tokens for job: job_1520871396884_0011
INFO : The url to track the job:
http://myhost:8088/proxy/application_1520871396884_0011/
INFO : Starting Job = job_1520871396884_0011, Tracking URL =
http://myhost:8088/proxy/application_1520871396884_0011/
INFO : Kill Command = /usr/lib/hadoop/bin/hadoop job -kill
job_1520871396884_0011
INFO : Hadoop job information for Stage-1: number of mappers: 9; number of
reducers: 1
INFO : 2018-03-12 13:27:32,281 Stage-1 map = 0%, reduce = 0%
INFO : 2018-03-12 13:27:51,424 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
8.69 sec
INFO : 2018-03-12 13:27:53,520 Stage-1 map = 70%, reduce = 0%, Cumulative CPU
91.66 sec
INFO : 2018-03-12 13:27:54,553 Stage-1 map = 85%, reduce = 0%, Cumulative CPU
93.15 sec
INFO : 2018-03-12 13:27:55,586 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 97.5 sec
INFO : 2018-03-12 13:28:01,793 Stage-1 map = 100%, reduce = 100%, Cumulative
CPU 100.0 sec
INFO : MapReduce Total cumulative CPU time: 1 minutes 40 seconds 0 msec
INFO : Ended Job = job_1520871396884_0011
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 9 Reduce: 1 Cumulative CPU: 100.0 sec HDFS
Read: 2317486328 HDFS Write: 10 SUCCESS
INFO : Total MapReduce CPU Time Spent: 1 minutes 40 seconds 0 msec
INFO : Completed executing
command(queryId=hive_20180312132727_5a75ce57-53e5-4ab2-9b97-31f8b3502969); Time
taken: 36.666 seconds
INFO : OK
-- Result: 110075425
{code}
{code:java|title=select count(1) from ngram2}
INFO : Compiling
command(queryId=hive_20180312133030_804a096b-f5d8-46c8-a255-4a958e6d2f0d):
select count(1) from ngram2
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
type:bigint, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20180312133030_804a096b-f5d8-46c8-a255-4a958e6d2f0d); Time
taken: 0.046 seconds
INFO : Executing
command(queryId=hive_20180312133030_804a096b-f5d8-46c8-a255-4a958e6d2f0d):
select count(1) from ngram2
INFO : Query ID = hive_20180312133030_804a096b-f5d8-46c8-a255-4a958e6d2f0d
INFO : Total jobs = 1
INFO : Launching Job 1 out of 1
INFO : Starting task
[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[
https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395857#comment-16395857
]
Hive QA commented on HIVE-17990:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} standalone-metastore: The patch generated 148 new +
1318 unchanged - 5 fixed = 1466 total (was 1323) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 285 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9604/dev-support/hive-personality.sh
|
| git revision | master / 0fe3879 |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9604/yetus/patch-mvninstall-itests_hcatalog-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9604/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9604/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9604/yetus/patch-asflicense-problems.txt
|
| modules | C: itests/hcatalog-unit standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Thrift and DB storage for Schema Registry objects
> -
>
> Key: HIVE-17990
> URL: https://issues.apache.org/jira/browse/HIVE-17990
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Labels: pull-request-available
> Attachments: Adding-Schema-Registry-to-Metastore.pdf,
> HIVE-17990.2.patch, HIVE-17990.3.patch, HIVE-17990.patch
>
>
> This JIRA tracks changes to Thrift, RawStore, and DB scripts to support
> objects in the Schema Registry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395837#comment-16395837
]
Aihua Xu commented on HIVE-14792:
-
Thanks [~mithun] for the code review.
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
> Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch,
> HIVE-14792.4.patch, HIVE-14792.5.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395833#comment-16395833
]
Mithun Radhakrishnan commented on HIVE-14792:
-
Ah, I see what you've done there! +1.
I applied your changes in a local setup, and sampled the failing tests. It
looks like you've sorted them out very well.
Thank you for working on this, [~aihuaxu]!
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
> Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch,
> HIVE-14792.4.patch, HIVE-14792.5.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395823#comment-16395823
]
Naveen Gangam commented on HIVE-18778:
--
how are things on your end? Thanks
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395821#comment-16395821
]
Naveen Gangam commented on HIVE-18778:
--
[~daijy] I could only attach files if I were the owner of the jira. So I had to
temporarily flip the ownership of this jira.
I have attached 3 separate patches with the names suffixed by the Spark Driver
names. Hopefully no one checks in any changes before we are able to commit
this. I had to do them in small batches as the running them in single run kept
failing.
I will get to the EncryptedHDFS ones later today or tomorrow.
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-18778:
-
Attachment: HIVE-18788_SparkNegative.patch
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-18778:
-
Attachment: HIVE-18788_SparkPerf.patch
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam reassigned HIVE-18778:
Assignee: Daniel Dai (was: Naveen Gangam)
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-18778:
-
Attachment: HIVE-18778-SparkPositive.patch
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam reassigned HIVE-18778:
Assignee: Naveen Gangam (was: Daniel Dai)
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-18778.1.patch, HIVE-18778.2.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395812#comment-16395812 ] Prasanth Jayachandran commented on HIVE-18907: -- lgtm, +1 > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395809#comment-16395809
]
Hive QA commented on HIVE-18343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914092/HIVE-18343.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 103 failed/errored test(s), 13356 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817
[ https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395806#comment-16395806 ] Jason Dere commented on HIVE-18907: --- [~prasanth_j] [~ekoifman] any additional feedback? > Create utility to fix acid key index issue from HIVE-18817 > -- > > Key: HIVE-18907 > URL: https://issues.apache.org/jira/browse/HIVE-18907 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, > HIVE-18907.3.patch > > > While HIVE-18817 will create new ORC Acid files from hitting the > ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will > still cause this issue. If there are delta directories then one way to > generate new files is to perform a major compaction. But this does not work > if there are no delta directories for the table/partition. > Add a tool to fix the Acid ORC files directly in the case that a compaction > cannot be performed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.
[
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395797#comment-16395797
]
Aihua Xu commented on HIVE-14792:
-
[~mithun]. Can you take a look quick look at the new patch to see if it makes
sense? Thanks.
> AvroSerde reads the remote schema-file at least once per mapper, per table
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
> Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch,
> HIVE-14792.4.patch, HIVE-14792.5.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties)
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
> schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
> // Get column names and sort order
> columnNames = Arrays.asList(columnNameProperty.split(","));
> columnTypes =
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
> schema = getSchemaFromCols(properties, columnNames, columnTypes,
> columnCommentProperty);
>
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
> schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized
> (i.e. at least once per mapper), the schema file is read remotely. For
> queries with thousands of mappers, this leads to a stampede to the handful
> (3?) datanodes that host the schema-file. In the best case, this causes
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the
> size-limit on table-parameters. The typical size of the Avro-schema file is
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made
> available as part of table-properties (but not serialized into the
> metastore), the downstream logic will remain largely intact. I have a patch
> that does this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16855) org.apache.hadoop.hive.ql.exec.mr.HashTableLoader Improvements
[
https://issues.apache.org/jira/browse/HIVE-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16855:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~belugabehr] for the contribution!
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader Improvements
> --
>
> Key: HIVE-16855
> URL: https://issues.apache.org/jira/browse/HIVE-16855
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.1.1, 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16855.1.patch, HIVE-16855.2.patch,
> HIVE-16855.3.patch
>
>
> # Improve (Simplify) Logging
> # Remove custom buffer size for {{BufferedInputStream}} and instead rely on
> JVM default which is often larger these days (8192)
> # Simplify looping logic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395771#comment-16395771
]
BELUGA BEHR commented on HIVE-18343:
Only failure was checkstyle, however while there still exists one check-style
issue, I'm not sure how to best address it, but I fixed 43 issues. Please
consider for submission as is.
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18884:
---
Description:
https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
The current logging is:
{code}
2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Trying to connect to metastore with URI
thrift://host.company.com:9083
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Connected to metastore.
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore, current connections: 2
{code}
Please simplify to something like:
{code}
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to the Metastore Server (URI
thrift://host.company.com:9083), current connections: 2
... or ...
2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Failed to connect to the Metastore Server (URI
thrift://host.company.com:9083)
{code}
was:
https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
The current logging is:
{code}
2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Trying to connect to metastore with URI
thrift://host.company.com:9083
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Connected to metastore.
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore, current connections: 2
{code}
Please simplify to something like:
{code}
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore (URI
thrift://host.company.com:9083), current connections: 2
... or ...
2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Failed to connect to the Metastore Server (URI
thrift://host.company.com:9083)
{code}
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to the Metastore Server (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Server (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18884) Simplify Logging in Hive Metastore Client
[
https://issues.apache.org/jira/browse/HIVE-18884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-18884:
---
Description:
https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
The current logging is:
{code}
2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Trying to connect to metastore with URI
thrift://host.company.com:9083
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Connected to metastore.
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore, current connections: 2
{code}
Please simplify to something like:
{code}
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore (URI
thrift://host.company.com:9083), current connections: 2
... or ...
2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Failed to connect to the Metastore Server (URI
thrift://host.company.com:9083)
{code}
was:
https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
The current logging is:
{code}
2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Trying to connect to metastore with URI
thrift://host.company.com:9083
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Connected to metastore.
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore, current connections: 2
{code}
Please simplify to something like:
{code}
2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Opened a connection to metastore (URI
thrift://host.company.com:9083), current connections: 2
... or ...
2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
Thread-65]: Failed to connect to the Metastore Sserver (URI
thrift://host.company.com:9083)
{code}
> Simplify Logging in Hive Metastore Client
> -
>
> Key: HIVE-18884
> URL: https://issues.apache.org/jira/browse/HIVE-18884
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> The current logging is:
> {code}
> 2018-02-26 07:02:44,883 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Trying to connect to metastore with URI
> thrift://host.company.com:9083
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Connected to metastore.
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore, current connections: 2
> {code}
> Please simplify to something like:
> {code}
> 2018-02-26 07:02:44,892 INFO hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Opened a connection to metastore (URI
> thrift://host.company.com:9083), current connections: 2
> ... or ...
> 2018-02-26 07:02:44,892 ERROR hive.metastore: [HiveServer2-Handler-Pool:
> Thread-65]: Failed to connect to the Metastore Server (URI
> thrift://host.company.com:9083)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[
https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395758#comment-16395758
]
Sahil Takiar commented on HIVE-18922:
-
Is this just when {{MERGE INTO}} queries are run? This might be specific to
Hive-ACID then.
> Hive is not cleaning up staging directories
>
>
> Key: HIVE-18922
> URL: https://issues.apache.org/jira/browse/HIVE-18922
> Project: Hive
> Issue Type: Bug
>Reporter: Anant Mittal
>Priority: Major
>
> Hive is creating hdfs folders with format
> /.hive-staging_hive__-xx/-ext-x
> These are not being cleaned up even after long duration. The folder is used
> to load to the table. Example:
> Loading data to table default.tablename from
> hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1
>
> This might be covered to some extent by HIVE-11940 but, want to make sure all
> cases are addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395755#comment-16395755 ] Anant Mittal commented on HIVE-18922: - [~stakiar] This was noticed in my setup every time a MERGE INTO query is used. I am not aware whether other cases would cause the same to happen in some specific scenario. > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18343) Remove LinkedList from ColumnStatsSemanticAnalyzer.java
[
https://issues.apache.org/jira/browse/HIVE-18343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395746#comment-16395746
]
Hive QA commented on HIVE-18343:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 2 unchanged - 43 fixed
= 3 total (was 45) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9603/dev-support/hive-personality.sh
|
| git revision | master / 53df7e8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9603/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9603/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9603/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove LinkedList from ColumnStatsSemanticAnalyzer.java
> ---
>
> Key: HIVE-18343
> URL: https://issues.apache.org/jira/browse/HIVE-18343
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-18343.1.patch, HIVE-18343.2.patch,
> HIVE-18343.3.patch
>
>
> Remove {{LinkedList}} in favor of {{ArrayList}} for class
> {{org.apache.hadoop.hive.ql.parse.ColumnStatsSemanticAnalyzer}}.
> {quote}
> The size, isEmpty, get, set, iterator, and listIterator operations run in
> constant time. The add operation runs in amortized constant time, that is,
> adding n elements requires O\(n\) time. All of the other operations run in
> linear time (roughly speaking). *The constant factor is low compared to that
> for the LinkedList implementation.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18922) Hive is not cleaning up staging directories
[ https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395744#comment-16395744 ] Sahil Takiar commented on HIVE-18922: - [~infinitymittal] can you provide some examples of when this is happening? Do you have an easy way to re-produce this? > Hive is not cleaning up staging directories > > > Key: HIVE-18922 > URL: https://issues.apache.org/jira/browse/HIVE-18922 > Project: Hive > Issue Type: Bug >Reporter: Anant Mittal >Priority: Major > > Hive is creating hdfs folders with format > /.hive-staging_hive__-xx/-ext-x > These are not being cleaned up even after long duration. The folder is used > to load to the table. Example: > Loading data to table default.tablename from > hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1 > > This might be covered to some extent by HIVE-11940 but, want to make sure all > cases are addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing
[
https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395710#comment-16395710
]
Hive QA commented on HIVE-18877:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12914087/HIVE-18877.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12959 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
