[jira] [Commented] (HIVE-18411) Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader
[
https://issues.apache.org/jira/browse/HIVE-18411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326838#comment-16326838
]
Hive QA commented on HIVE-18411:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 6938fca |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8622/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader
> -
>
> Key: HIVE-18411
> URL: https://issues.apache.org/jira/browse/HIVE-18411
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
>Priority: Critical
> Attachments: HIVE-18411.001.patch
>
>
> ColumnVector should be initialized to the default size at the begin of
> readBatch(), otherwise, ArrayIndexOutOfBoundsException will be thrown because
> the size of ColumnVector may be updated in the last readBatch().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326788#comment-16326788 ] Gopal V commented on HIVE-17896: Thanks Teddy, I will look into the new patch. > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18411) Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader
[
https://issues.apache.org/jira/browse/HIVE-18411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326773#comment-16326773
]
Hive QA commented on HIVE-18411:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905208/HIVE-18411.001.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8621/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8621/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8621/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-16 05:27:50.282
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8621/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-16 05:27:50.286
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 6938fca HIVE-18443: Ensure git gc finished in ptest prep phase
before copying repo (Adam Szita, via Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 6938fca HIVE-18443: Ensure git gc finished in ptest prep phase
before copying repo (Adam Szita, via Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-16 05:27:56.796
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8621/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedListColumnReader.java:
does not exist in index
error:
a/ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedListColumnReader.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: protoc version: 250, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protoc8586241245650140697.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-common: Failed to resolve
dependencies for one or more projects in the reactor. Reason: Missing:
[ERROR] --
[ERROR] 1) net.minidev:json-smart:jar:2.3-SNAPSHOT
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=net.minidev -DartifactId=json-smart
-Dversion=2.3-SNAPSHOT -Dpackaging=jar -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file
there:
[ERROR] mvn
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.02.patch * Removing validation of query parallelism. * Changing alloc fraction validation so that, except for root level, sum(child alloc fraction) < 1. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18411) Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader
[ https://issues.apache.org/jira/browse/HIVE-18411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326765#comment-16326765 ] Colin Ma commented on HIVE-18411: - Thanks for review, I'll trigger the pre-commit. > Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader > - > > Key: HIVE-18411 > URL: https://issues.apache.org/jira/browse/HIVE-18411 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Critical > Attachments: HIVE-18411.001.patch > > > ColumnVector should be initialized to the default size at the begin of > readBatch(), otherwise, ArrayIndexOutOfBoundsException will be thrown because > the size of ColumnVector may be updated in the last readBatch(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326763#comment-16326763
]
Vineet Garg commented on HIVE-16605:
Attached is first patch which supports NOT NULL constraint enforcement on
INSERT statements. Note that this patch is not yet fully tested therefore is
not yet ready to be committed.
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16605:
---
Status: Patch Available (was: Open)
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18411) Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader
[ https://issues.apache.org/jira/browse/HIVE-18411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326762#comment-16326762 ] Ferdinand Xu commented on HIVE-18411: - Sorry for my late response. LGTM +1. Could you retrigger the precommit to see whether it failed new cases on newer base? > Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader > - > > Key: HIVE-18411 > URL: https://issues.apache.org/jira/browse/HIVE-18411 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma >Priority: Critical > Attachments: HIVE-18411.001.patch > > > ColumnVector should be initialized to the default size at the begin of > readBatch(), otherwise, ArrayIndexOutOfBoundsException will be thrown because > the size of ColumnVector may be updated in the last readBatch(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-16605:
---
Attachment: HIVE-16605.1.patch
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg reassigned HIVE-16605:
--
Assignee: Vineet Garg
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326740#comment-16326740 ] Teddy Choi commented on HIVE-17896: --- [~gopalv], I fixed the DPP optimizer bug and uploaded it on RB. > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18386) Create dummy materialized views registry and make it configurable
[ https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326739#comment-16326739 ] Jesus Camacho Rodriguez commented on HIVE-18386: The patch also fixes issues with the initialization of the registry for CliDriver (and tests using it). Still pending fix for SessionState not being initialized prior to the registry, which causes issues with authorization. I will upload a new patch shortly. > Create dummy materialized views registry and make it configurable > - > > Key: HIVE-18386 > URL: https://issues.apache.org/jira/browse/HIVE-18386 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18386.01.patch > > > HiveMaterializedViewsRegistry keeps the materialized views plans in memory to > have quick access when queries are planned. For debugging purposes, we will > create a dummy materialized views registry that forwards all calls to > metastore and make the choice configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326735#comment-16326735 ] Jason Dere commented on HIVE-18401: --- The q.out actually did need to be updated because of this Jira, but I think HIVE-18385 may have been preventing that. Looks like HIVE-18385 was not occurring in the pre-commit tests, i think it was a timing issue. > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-17896: -- Attachment: HIVE-17896.6.patch > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, > HIVE-17896.4.patch, HIVE-17896.5.patch, HIVE-17896.6.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable
[ https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18386: --- Attachment: HIVE-18386.01.patch > Create dummy materialized views registry and make it configurable > - > > Key: HIVE-18386 > URL: https://issues.apache.org/jira/browse/HIVE-18386 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18386.01.patch > > > HiveMaterializedViewsRegistry keeps the materialized views plans in memory to > have quick access when queries are planned. For debugging purposes, we will > create a dummy materialized views registry that forwards all calls to > metastore and make the choice configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable
[ https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18386: --- Attachment: (was: HIVE-18386.patch) > Create dummy materialized views registry and make it configurable > - > > Key: HIVE-18386 > URL: https://issues.apache.org/jira/browse/HIVE-18386 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HiveMaterializedViewsRegistry keeps the materialized views plans in memory to > have quick access when queries are planned. For debugging purposes, we will > create a dummy materialized views registry that forwards all calls to > metastore and make the choice configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable
[ https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18386: --- Status: Patch Available (was: In Progress) > Create dummy materialized views registry and make it configurable > - > > Key: HIVE-18386 > URL: https://issues.apache.org/jira/browse/HIVE-18386 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HiveMaterializedViewsRegistry keeps the materialized views plans in memory to > have quick access when queries are planned. For debugging purposes, we will > create a dummy materialized views registry that forwards all calls to > metastore and make the choice configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326717#comment-16326717 ] Vihang Karajgaonkar edited comment on HIVE-18401 at 1/16/18 3:25 AM: - Oh okay. I saw the diffs to the q.out from this patch and thought it was related. Thanks for the pointer. was (Author: vihangk1): Oh okay. I thought the diffs to the q.out from this patch and thought it was related. Thanks for the pointer. > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326717#comment-16326717 ] Vihang Karajgaonkar commented on HIVE-18401: Oh okay. I thought the diffs to the q.out from this patch and thought it was related. Thanks for the pointer. > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-17434) Using "add jar " from viewFs always occurred hdfs mismatch error
[ https://issues.apache.org/jira/browse/HIVE-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17434 started by Bang Xiao. > Using "add jar " from viewFs always occurred hdfs mismatch error > > > Key: HIVE-17434 > URL: https://issues.apache.org/jira/browse/HIVE-17434 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 1.2.2, 1.2.3 >Reporter: shenxianqiang >Assignee: Bang Xiao >Priority: Minor > Fix For: 1.2.1, 1.2.3 > > Attachments: HIVE-17434.2-branch-1.2.patch, HIVE-17434.patch > > > add jar viewfs://nsX//lib/common.jar > always occure mismatch error -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326713#comment-16326713 ] Jason Dere commented on HIVE-18385: --- Stats seem to be different between MacOS and Linux for some reason. Re-attaching patch after updating golden file on Linux. > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18385.1.patch, HIVE-18385.2.patch, hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at >
[jira] [Updated] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException
[ https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18385: -- Attachment: HIVE-18385.2.patch > mergejoin fails with java.lang.IllegalStateException > > > Key: HIVE-18385 > URL: https://issues.apache.org/jira/browse/HIVE-18385 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18385.1.patch, HIVE-18385.2.patch, hive.log > > > mergejoin test fails with java.lang.IllegalStateException when run in > MiniLlapLocal. > This is the query for which it fails, > [ERROR] TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed > with error code = 2 running " > select count(*) from tab a join tab_part b on a.key = b.key join src1 c on > a.value = c.value" fname=mergejoin.q > This is the stack trace, > failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, > vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: > ORC split generation failed with exception: java.lang.IllegalStateException: > Failed to retrieve dynamic value for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499) > at > org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684) > at > org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269) > at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253) > at > com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) > at > com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) > at > com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Failed to retrieve dynamic value for > RS_12_a_key_min > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778) > ... 17 more > Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value > for RS_12_a_key_min > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97) > at > org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93) > at > org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463) > at > org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476) > at >
[jira] [Commented] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326683#comment-16326683 ] Deepak Jaiswal commented on HIVE-18401: --- [~vihangk1] the test has been failing for much longer. I filed a Jira which is already in works, https://issues.apache.org/jira/browse/HIVE-18385 > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18450) Support TABLE function in CBO
[ https://issues.apache.org/jira/browse/HIVE-18450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-18450: --- Attachment: HIVE-18450.02.patch > Support TABLE function in CBO > - > > Key: HIVE-18450 > URL: https://issues.apache.org/jira/browse/HIVE-18450 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18450.01.patch, HIVE-18450.02.patch > > > Follow-up of HIVE-18416 to support TABLE function in CBO. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[ https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326615#comment-16326615 ] Vihang Karajgaonkar commented on HIVE-18372: +1 LGTM. Thanks for the changes [~pvary] > Create testing infra to test different HMS instances > > > Key: HIVE-18372 > URL: https://issues.apache.org/jira/browse/HIVE-18372 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch, > HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch > > > Since there will be multiple tests, it would be good to have a good > infrastructure to help creating those faster, easier. > This patch will also include the test cases for the Database related methods > to showcase the infra -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18449) Add configurable policy for choosing the HMS URI from hive.metastore.uris
[
https://issues.apache.org/jira/browse/HIVE-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326607#comment-16326607
]
Vihang Karajgaonkar commented on HIVE-18449:
possibly related to what HIVE-18347 is doing. It introduces a pluggable way to
resolve the URLs for the HMS client.
> Add configurable policy for choosing the HMS URI from hive.metastore.uris
> -
>
> Key: HIVE-18449
> URL: https://issues.apache.org/jira/browse/HIVE-18449
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>Priority: Major
>
> HIVE-10815 added logic to randomly choose a HMS URI from
> {{hive.metastore.uris}}. It would be nice if there was a configurable policy
> that determined how a URI is chosen from this list - e.g. one option can be
> to randomly pick a URI, another option can be to choose the first URI in the
> list (which was the behavior prior to HIVE-10815).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326606#comment-16326606 ] Vihang Karajgaonkar commented on HIVE-18347: Hi [~szehon] I like the idea of having a pluggable way to resolve the URIs. I think this is similar to what HIVE-18449 requires. May be the existing mechanism of randomizing the order of URIs could also be implemented as a URIResolverHook implementation and then the code would look a lot cleaner. Regarding Consul support, I am not sure what Consul is and I took a quick look at it. It seems to be that it is open sourced under Mozilla Public License so not sure if there are any licensing complications while integrating with it. Perhaps the Consul URI resolver hook implementation from this patch could be separate from the Hive source code and can be just added as a custom jar in your environment? Off-course, I am not a 100% sure either so may be other senior folks here may be some definitive answer on adding such code for outside tools. Or you could always email on the dev list. > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326602#comment-16326602
]
Vihang Karajgaonkar edited comment on HIVE-18323 at 1/15/18 10:20 PM:
--
Fixed check style errors and vectorized_parquet_types test. The other tests
which are failing are un-related.
was (Author: vihangk1):
Fixed check style errors and parquet_vectorized_types.q test.
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18323.02.patch, HIVE-18323.03.patch,
> HIVE-18323.04.patch, HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326602#comment-16326602
]
Vihang Karajgaonkar commented on HIVE-18323:
Fixed check style errors and parquet_vectorized_types.q test.
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18323.02.patch, HIVE-18323.03.patch,
> HIVE-18323.04.patch, HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18323:
---
Attachment: HIVE-18323.04.patch
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18323.02.patch, HIVE-18323.03.patch,
> HIVE-18323.04.patch, HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18401) load data should rename files consistent with insert statements (bucketed tables only) Part3
[ https://issues.apache.org/jira/browse/HIVE-18401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326601#comment-16326601 ] Vihang Karajgaonkar commented on HIVE-18401: Hi [~djaiswal] mergejoin.q is failing since this patch. Do we need to update the mergejoin.q.out? Thanks! > load data should rename files consistent with insert statements (bucketed > tables only) Part3 > > > Key: HIVE-18401 > URL: https://issues.apache.org/jira/browse/HIVE-18401 > Project: Hive > Issue Type: Sub-task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18401.1.patch, HIVE-18401.2.patch, > HIVE-18401.3.patch, HIVE-18401.4.patch, HIVE-18401.5.patch, HIVE-18401.6.patch > > > As part of this effort, this JIRA tracks updating tests which use load data > files srcbucket20 etc > Following is the mapping of new data files vs old data files > bmj/00_0 —> srcbucket20.txt > bmj/01_0 —> srcbucket21.txt > bmj/02_0 —> srcbucket22.txt > bmj/03_0 —> srcbucket23.txt > bmj1/01_0 —> srcbucket22.txt > bmj2/00_0 —> srcbucket22.txt > bmj2/01_0 —> srcbucket23.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-15436) Enhancing metastore APIs to retrieve only materialized views
[
https://issues.apache.org/jira/browse/HIVE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-15436:
--
Comment: was deleted
(was: Hit some error after the patch, seems we need the following change:
{code:java}
---
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
+++
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
@@ -408,8 +408,8 @@ public Database getDatabase(String dbName) throws
MetaException{
queryParams, pms.toArray(), queryText));
if (!sqlResult.isEmpty()) {
- for (Object[] line : sqlResult) {
- ret.add(extractSqlString(line[0]));
+ for (Object line : sqlResult) {
+ ret.add(extractSqlString(line));
}
}
return ret;
{code})
> Enhancing metastore APIs to retrieve only materialized views
>
>
> Key: HIVE-15436
> URL: https://issues.apache.org/jira/browse/HIVE-15436
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15436.patch
>
>
> Enhancing metastore APIs such that, instead of returning all tables, it can
> return only:
> - views
> - materialized views
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15436) Enhancing metastore APIs to retrieve only materialized views
[
https://issues.apache.org/jira/browse/HIVE-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326584#comment-16326584
]
Daniel Dai commented on HIVE-15436:
---
Hit some error after the patch, seems we need the following change:
{code:java}
---
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
+++
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
@@ -408,8 +408,8 @@ public Database getDatabase(String dbName) throws
MetaException{
queryParams, pms.toArray(), queryText));
if (!sqlResult.isEmpty()) {
- for (Object[] line : sqlResult) {
- ret.add(extractSqlString(line[0]));
+ for (Object line : sqlResult) {
+ ret.add(extractSqlString(line));
}
}
return ret;
{code}
> Enhancing metastore APIs to retrieve only materialized views
>
>
> Key: HIVE-15436
> URL: https://issues.apache.org/jira/browse/HIVE-15436
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15436.patch
>
>
> Enhancing metastore APIs such that, instead of returning all tables, it can
> return only:
> - views
> - materialized views
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326543#comment-16326543
]
Vihang Karajgaonkar commented on HIVE-18323:
Yes, most test failures are unrelated. I will take relook at the ones which
look related. Also, will submit the patch again with checkstyle fixes so that
yetus is happy.
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18323.02.patch, HIVE-18323.03.patch,
> HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.3.patch
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18296) Document hive.exec.local.scratchdir
[
https://issues.apache.org/jira/browse/HIVE-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326449#comment-16326449
]
BELUGA BEHR commented on HIVE-18296:
[~leftylev] Thanks!
> Document hive.exec.local.scratchdir
> ---
>
> Key: HIVE-18296
> URL: https://issues.apache.org/jira/browse/HIVE-18296
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Lefty Leverenz
>Priority: Minor
>
> Document configuration variable {{hive.exec.local.scratchdir}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-18296) Document hive.exec.local.scratchdir
[
https://issues.apache.org/jira/browse/HIVE-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR resolved HIVE-18296.
Resolution: Fixed
> Document hive.exec.local.scratchdir
> ---
>
> Key: HIVE-18296
> URL: https://issues.apache.org/jira/browse/HIVE-18296
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Lefty Leverenz
>Priority: Minor
>
> Document configuration variable {{hive.exec.local.scratchdir}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18323) Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
[
https://issues.apache.org/jira/browse/HIVE-18323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326410#comment-16326410
]
Aihua Xu commented on HIVE-18323:
-
[~vihangk1] The tests don't seem related but can you double check? The patch
looks good to me. Thanks for adding the unit tests. +1.
> Vectorization: add the support of timestamp in
> VectorizedPrimitiveColumnReader for parquet
> --
>
> Key: HIVE-18323
> URL: https://issues.apache.org/jira/browse/HIVE-18323
> Project: Hive
> Issue Type: Sub-task
> Components: Vectorization
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18323.02.patch, HIVE-18323.03.patch,
> HIVE-18323.1.patch
>
>
> {noformat}
> CREATE TABLE `t1`(
> `ts` timestamp,
> `s1` string)
> STORED AS PARQUET;
> set hive.vectorized.execution.enabled=true;
> SELECT * from t1 SORT BY s1;
> {noformat}
> This query will throw exception since timestamp is not supported here yet.
> {noformat}
> Caused by: java.io.IOException: java.io.IOException: Unsupported type:
> optional int96 ts
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[ https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326398#comment-16326398 ] Peter Vary commented on HIVE-18372: --- Test failures are not related. Any further comments [~vihangk1], [~alangates]? Thanks, Peter > Create testing infra to test different HMS instances > > > Key: HIVE-18372 > URL: https://issues.apache.org/jira/browse/HIVE-18372 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch, > HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch > > > Since there will be multiple tests, it would be good to have a good > infrastructure to help creating those faster, easier. > This patch will also include the test cases for the Database related methods > to showcase the infra -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326393#comment-16326393
]
Hive QA commented on HIVE-18372:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906114/HIVE-18372.6.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11619 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=255)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8620/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8620/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8620/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12906114 - PreCommit-HIVE-Build
> Create testing infra to test different HMS instances
>
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch,
> HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods
> to showcase the infra
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326349#comment-16326349
]
Hive QA commented on HIVE-18372:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 6938fca |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8620/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create testing infra to test different HMS instances
>
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch,
> HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods
> to showcase the infra
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18347) Allow dynamic lookup of Hive Metastores via Consul
[ https://issues.apache.org/jira/browse/HIVE-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326348#comment-16326348 ] Szehon Ho commented on HIVE-18347: -- [~alangates] [~vihangk1] any thoughts on whether this is a useful contribution to hive? Hive deployment in our organization (Criteo) uses open source tool Consul to do service discovery/state for Hive Metastores, but I am not sure the community guidelines on adding support for outside tools like this. Though this patch does allow pluggable to other service discovery mechanisms. > Allow dynamic lookup of Hive Metastores via Consul > -- > > Key: HIVE-18347 > URL: https://issues.apache.org/jira/browse/HIVE-18347 > Project: Hive > Issue Type: New Feature > Components: Metastore >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-18347.1.patch, HIVE-18347.2.patch, > HIVE-18347.3.patch > > > In our organization, we have deployed HiveMetastore and HiveServer2 on Mesos > as dynamic services for scalability and flexibility. > In this architecture, we would like to allow HiveServer2 to dynamically load > balance between Metastores (which may be scaled up and down or to different > nodes) for different requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-12338) Add webui to HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326339#comment-16326339 ] Szehon Ho commented on HIVE-12338: -- Hey not yet, but I think it is pretty easy to do. I had made HIVE-13457 but have not had time to do this yet. > Add webui to HiveServer2 > > > Key: HIVE-12338 > URL: https://issues.apache.org/jira/browse/HIVE-12338 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Major > Attachments: HIVE-12338.1.patch, HIVE-12338.2.patch, > HIVE-12338.3.patch, HIVE-12338.4.patch, hs2-conf.png, hs2-logs.png, > hs2-metrics.png, hs2-webui.png > > > A web ui for HiveServer2 can show some useful information such as: > > 1. Sessions, > 2. Queries that are executing on the HS2, their states, starting time, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326324#comment-16326324
]
Hive QA commented on HIVE-18372:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906114/HIVE-18372.6.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11198 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326268#comment-16326268
]
Hive QA commented on HIVE-18372:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 6938fca |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8619/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create testing infra to test different HMS instances
>
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
> Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch,
> HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods
> to showcase the infra
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances
[ https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326244#comment-16326244 ] Peter Vary commented on HIVE-18372: --- [~alangates]: Created the wiki page, and added IMetaStoreClient paragraph - feel free to criticize. English is not my native language :) [https://cwiki.apache.org/confluence/display/Hive/MetaStore+API+tests] > Create testing infra to test different HMS instances > > > Key: HIVE-18372 > URL: https://issues.apache.org/jira/browse/HIVE-18372 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch, > HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch > > > Since there will be multiple tests, it would be good to have a good > infrastructure to help creating those faster, easier. > This patch will also include the test cases for the Database related methods > to showcase the infra -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18372) Create testing infra to test different HMS instances
[ https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-18372: -- Attachment: HIVE-18372.6.patch Since HIVE-18355 is committed I can use the FunctionBuilder in the test too. > Create testing infra to test different HMS instances > > > Key: HIVE-18372 > URL: https://issues.apache.org/jira/browse/HIVE-18372 > Project: Hive > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-18372.2.patch, HIVE-18372.3.patch, > HIVE-18372.5.patch, HIVE-18372.6.patch, HIVE-18372.patch > > > Since there will be multiple tests, it would be good to have a good > infrastructure to help creating those faster, easier. > This patch will also include the test cases for the Database related methods > to showcase the infra -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326188#comment-16326188
]
Hive QA commented on HIVE-18442:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906094/HIVE-18442.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=255)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8618/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8618/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8618/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12906094 - PreCommit-HIVE-Build
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
>
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326165#comment-16326165
]
Oleksiy Sayankin commented on HIVE-16950:
-
Hi all.
Let's discuss some ideas about this issue. When I have implemented verification
for tables (check if table folder exists before creation of a table, and if yes
- throw an exception), I have faced some problems here:
# SemanticAnalyzer uses temp tables placed in one folder, so it starts to fail
in INSERT operations.
# Customers may want to create external tables with existing data/folders to
escape from data copying.
So I have added two exceptions: temporary and external tables may use the same
folder. Even in that case I have some tests failed (e.g.
org.apache.hive.hcatalog.api.repl.commands.TestCommands.testBasicReplEximCommands).
Import command does not work with the fix for tables:
{code:java}
hive> export table student to '/user/mysudent';
hive> import table imported_student from '/user/mysudent';
{code}
results with
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:Failed
to create table. Table directory already exists
{code}
So my suggestion is to implement fix for databases but not for the tables. Any
ideas?
> Dropping hive database/table which was created explicitly in default database
> location, deletes all databases data from default database location
> -
>
> Key: HIVE-16950
> URL: https://issues.apache.org/jira/browse/HIVE-16950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Rahul Kalgunde
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Attachments: HIVE-16950.1.patch, HIVE-16950.2.patch
>
>
> When database/table is created explicitly pointing to the default location,
> dropping the database/table deletes all the data associated with the all
> databases/tables.
> Steps to replicate:
> in below e.g. dropping table test_db2 also deletes data of test_db1 where as
> metastore still contains test_db1
> hive> create database test_db1;
> OK
> Time taken: 4.858 seconds
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.599 seconds, Fetched: 1 row(s)
> hive> create database test_db2 location '/apps/hive/warehouse' ;
> OK
> Time taken: 1.457 seconds
> hive> describe database test_db2;
> OK
> test_db2
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse rootUSER
> Time taken: 0.582 seconds, Fetched: 1 row(s)
> hive> drop database test_db2;
> OK
> Time taken: 1.317 seconds
> hive> dfs -ls /apps/hive/warehouse;
> ls: `/apps/hive/warehouse': No such file or directory
> Command failed with exit code = 1
> Query returned non-zero code: 1, cause: null
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.629 seconds, Fetched: 1 row(s)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326142#comment-16326142
]
Hive QA commented on HIVE-18442:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 6938fca |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8618/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
>
[jira] [Commented] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326133#comment-16326133
]
Rui Li commented on HIVE-18442:
---
I didn't add a qtest for this because we have hive-exec.jar in driver's extra
class path in the test, which can solve the issue. But since we can't expect
this in real deployment, code change is still needed.
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-18442:
--
Status: Patch Available (was: Open)
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-18442:
--
Attachment: HIVE-18442.1.patch
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326105#comment-16326105
]
Zoltan Haindrich commented on HIVE-18445:
-
I was looking into whether this feature works for producing test cases for
reoptimization...and I've probably just bumped into the reason this test have
changed after HIVE-13567:
[https://github.com/apache/hive/blob/6938fcabf0944817fed8241c48b57fb3f5d98c69/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java#L133
which is a feature introduced in HIVE-1830
so probably setting {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} to
some low value would probably re-introduce the backuptasks
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17982) Move metastore specific itests
[ https://issues.apache.org/jira/browse/HIVE-17982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326096#comment-16326096 ] Peter Vary commented on HIVE-17982: --- Thanks for the patch [~alangates]! +1 > Move metastore specific itests > -- > > Key: HIVE-17982 > URL: https://issues.apache.org/jira/browse/HIVE-17982 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17982.2.patch, HIVE-17982.patch > > > There are a number of tests in itests/hive-unit/.../metastore that are > metastore specific. I suspect they were initially placed in itests only > because the metastore pulling in a few plugins from ql. > Given that we need to be able to release the metastore separately, we need to > be able to test it completely as a standalone entity. So I propose to move a > number of the itests over into standalone-metastore. I will only move tests > that are isolated to the metastore. Anything that tests wider functionality > I plan to leave in itests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18214) Flaky test: TestSparkClient
[
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326077#comment-16326077
]
Peter Vary commented on HIVE-18214:
---
+1
Thanks for the explanation [~stakiar]!
> Flaky test: TestSparkClient
> ---
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18214.1.patch, HIVE-18214.2.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A
> new {{JavaSparkContext}} is created for each test that is run. There is a
> race condition where the {{RemoteDriver}} isn't given enough time to
> shutdown, so when the next test starts running it creates another
> {{JavaSparkContext}} which causes an exception like
> {{org.apache.spark.SparkException: Only one SparkContext may be running in
> this JVM (see SPARK-2243)}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18443) Ensure git gc finished in ptest prep phase before copying repo
[
https://issues.apache.org/jira/browse/HIVE-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326074#comment-16326074
]
Adam Szita commented on HIVE-18443:
---
Thanks for the review and commit Peter!
> Ensure git gc finished in ptest prep phase before copying repo
> --
>
> Key: HIVE-18443
> URL: https://issues.apache.org/jira/browse/HIVE-18443
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18443.0.patch
>
>
> In ptest's prep phase script first we checkout the latest Hive code from git,
> and then we make copy of its contents (along .git folder) for that will serve
> as Yetus' working directory.
> In some cases we can see errors such as
> {code}
> + cp -R . ../yetus
> cp: cannot stat ?./.git/gc.pid?: No such file or directory
> {code}
> e.g.
> [here|https://issues.apache.org/jira/browse/HIVE-18372?focusedCommentId=16321507=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16321507]
> This is caused by git running its gc feature in the background when our prep
> script has already started copying. In cases where gc finishes while cp is
> running, we'll get this error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18443) Ensure git gc finished in ptest prep phase before copying repo
[
https://issues.apache.org/jira/browse/HIVE-18443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-18443:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks [~szita] for your patch!
> Ensure git gc finished in ptest prep phase before copying repo
> --
>
> Key: HIVE-18443
> URL: https://issues.apache.org/jira/browse/HIVE-18443
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18443.0.patch
>
>
> In ptest's prep phase script first we checkout the latest Hive code from git,
> and then we make copy of its contents (along .git folder) for that will serve
> as Yetus' working directory.
> In some cases we can see errors such as
> {code}
> + cp -R . ../yetus
> cp: cannot stat ?./.git/gc.pid?: No such file or directory
> {code}
> e.g.
> [here|https://issues.apache.org/jira/browse/HIVE-18372?focusedCommentId=16321507=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16321507]
> This is caused by git running its gc feature in the background when our prep
> script has already started copying. In cases where gc finishes while cp is
> running, we'll get this error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18355) Add builder for metastore Thrift classes missed in the first pass - FunctionBuilder
[ https://issues.apache.org/jira/browse/HIVE-18355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-18355: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~alangates] for the review! > Add builder for metastore Thrift classes missed in the first pass - > FunctionBuilder > --- > > Key: HIVE-18355 > URL: https://issues.apache.org/jira/browse/HIVE-18355 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-18355.2.patch, HIVE-18355.patch > > > Add a FunctionBuilder class -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[
https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326039#comment-16326039
]
Hive QA commented on HIVE-18061:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906071/HIVE-18061.10.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 11560 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket4]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[leftsemijoin_mr]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[scriptfile1]
(batchId=180)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=214)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=255)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8617/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8617/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8617/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12906071 - PreCommit-HIVE-Build
> q.outs: be more selective with masking hdfs paths
> -
>
> Key: HIVE-18061
> URL: https://issues.apache.org/jira/browse/HIVE-18061
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch,
> HIVE-18061.03.patch, HIVE-18061.04.patch, HIVE-18061.05.patch,
> HIVE-18061.06.patch, HIVE-18061.07.patch, HIVE-18061.08.patch,
> HIVE-18061.09.patch, HIVE-18061.10.patch
>
>
> currently any line which contains a path which looks like an hdfs location is
> replaced with a "masked pattern was here"...
> it might be releavant to record these messages; since even an exception
> message might contain an hdfs location
> noticed in
> HIVE-18012
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18413) Grouping of an empty result set may only contain null values
[
https://issues.apache.org/jira/browse/HIVE-18413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326031#comment-16326031
]
Zoltan Haindrich commented on HIVE-18413:
-
[~ashutoshc] could you please take a look?
> Grouping of an empty result set may only contain null values
>
>
> Key: HIVE-18413
> URL: https://issues.apache.org/jira/browse/HIVE-18413
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18413.01.patch, HIVE-18413.02.patch,
> HIVE-18413.03.patch
>
>
> exposed by: HIVE-18359
> in case of vectorization, the summary row object was left as is (presumed
> null earlier); which may cause it to be inconsistent isNull conditions in
> .VectorHashKeyWrapperBatch
> issue happens only if:
> * vectorizable groupby
> * groupping set contains empty
> * non-trivial empty; mapper is run
> * groupping key is select ; with a type which is backed by a bytea; ex:string
> {code}
> set hive.vectorized.execution.enabled=true;
> create table tx2 (a integer,b integer,c integer,d double,u string,bi binary)
> stored as orc;
> insert into tx2 values
> (1,2,3,1.1,'x','b'),
> (3,2,3,1.1,'y','b');
> select sum(a),
> u,
> bi,
> 'asd',
> grouping(bi),
> 'NULL,1' as expected
> fromtx2
> where a=2
> group by a,u,bi grouping sets ( u, (), bi);
> {code}
> causes:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.assignRowColumn(VectorHashKeyWrapperBatch.java:1065)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:1134)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$800(VectorGroupByOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.close(VectorGroupByOperator.java:862)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1176)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:705)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> ... 16 more
> ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
> killedTasks:0, Vertex vertex_1515531021543_0001_12_01 [Reducer 2]
> killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:0
> 2018-01-09T12:50:30,611 DEBUG [01fdcefd-40b0-45a6-8e5b-b1cd14241088 main]
> ql.Driver: Shutting down query
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326030#comment-16326030 ] Zoltan Haindrich commented on HIVE-18445: - query plans look good to me ; I've see no changes in the select join type (other changes are expect after HIVE-13567) ; interestingly: adding these explains to the q file made the "Execution Error" messages go away somehow > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326021#comment-16326021
]
Zoltan Haindrich commented on HIVE-18445:
-
it seems like the test was added in HIVE-1642; with the intention to check for
the "backuptask" doing the job in case the first one fails...I think the test
should have ~3 BackupTask executionions in the q.out; but instead there is 1
In the light of the above; I've looked into when these {{BackupTask}} attempts
are changed - and it looks like after HIVE-13567 some of the "Execution
Error"-s are gone; probably the plans have changed for these queries
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18273) add LLAP-level counters for WM
[ https://issues.apache.org/jira/browse/HIVE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326018#comment-16326018 ] Harish Jaiprakash commented on HIVE-18273: -- +1 for the code. > add LLAP-level counters for WM > -- > > Key: HIVE-18273 > URL: https://issues.apache.org/jira/browse/HIVE-18273 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18273.01.patch, HIVE-18273.02.patch, > HIVE-18273.patch > > > On query fragment level (like IO counters) > time queued as guaranteed; > time running as guaranteed; > time running as speculative. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18314) qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
[
https://issues.apache.org/jira/browse/HIVE-18314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326004#comment-16326004
]
Zoltan Haindrich commented on HIVE-18314:
-
+1
> qtests: semijoin_hint.q breaks hybridgrace_hashjoin_2.q
>
>
> Key: HIVE-18314
> URL: https://issues.apache.org/jira/browse/HIVE-18314
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18314.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=semijoin_hint.q,hybridgrace_hashjoin_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[
https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326005#comment-16326005
]
Hive QA commented on HIVE-18061:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/util: The patch generated 1 new + 188 unchanged
- 6 fixed = 189 total (was 194) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 22df53b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8617/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: ql itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8617/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> q.outs: be more selective with masking hdfs paths
> -
>
> Key: HIVE-18061
> URL: https://issues.apache.org/jira/browse/HIVE-18061
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch,
> HIVE-18061.03.patch, HIVE-18061.04.patch, HIVE-18061.05.patch,
> HIVE-18061.06.patch, HIVE-18061.07.patch, HIVE-18061.08.patch,
> HIVE-18061.09.patch, HIVE-18061.10.patch
>
>
> currently any line which contains a path which looks like an hdfs location is
> replaced with a "masked pattern was here"...
> it might be releavant to record these messages; since even an exception
> message might contain an hdfs location
> noticed in
> HIVE-18012
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18309) qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
[
https://issues.apache.org/jira/browse/HIVE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326001#comment-16326001
]
Zoltan Haindrich commented on HIVE-18309:
-
+1; I've looked around a bit more; and HIVE-10530 have also disabled
{{hive.metastore.aggregate.stats.cache.enabled}} in some tests
> qtests: smb_mapjoin_19.q breaks bucketsortoptimize_insert_2.q
> -
>
> Key: HIVE-18309
> URL: https://issues.apache.org/jira/browse/HIVE-18309
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18309.01.patch
>
>
> {code}
> mvn install -q -am -pl itests/qtest -DskipSparkTests -pl itests/qtest
> -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=smb_mapjoin_19.q,bucketsortoptimize_insert_2.q
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18061) q.outs: be more selective with masking hdfs paths
[ https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-18061: Attachment: HIVE-18061.10.patch > q.outs: be more selective with masking hdfs paths > - > > Key: HIVE-18061 > URL: https://issues.apache.org/jira/browse/HIVE-18061 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch, > HIVE-18061.03.patch, HIVE-18061.04.patch, HIVE-18061.05.patch, > HIVE-18061.06.patch, HIVE-18061.07.patch, HIVE-18061.08.patch, > HIVE-18061.09.patch, HIVE-18061.10.patch > > > currently any line which contains a path which looks like an hdfs location is > replaced with a "masked pattern was here"... > it might be releavant to record these messages; since even an exception > message might contain an hdfs location > noticed in > HIVE-18012 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
