[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356612#comment-16356612
]
Colin Ma commented on HIVE-14171:
-
[~vihangk1], [~KaiXu], I think this NPE problem is fixed in HIVE-15718 and
HIVE-16465.
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356610#comment-16356610
]
Hive QA commented on HIVE-18350:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore: The patch generated 2 new + 391
unchanged - 1 fixed = 393 total (was 392) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 9 new + 204 unchanged - 4
fixed = 213 total (was 208) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
55s{color} | {color:red} root: The patch generated 11 new + 604 unchanged - 5
fixed = 615 total (was 609) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 16b8575 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9087/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9087/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9087/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9087/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9087/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch,
> HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.13.patch,
> HIVE-18350.14.patch, HIVE-18350.15.patch, HIVE-18350.16.patch,
> HIVE-18350.2.patch, HIVE-18350
[jira] [Commented] (HIVE-18238) Driver execution may not have configuration changing sideeffects
[
https://issues.apache.org/jira/browse/HIVE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356556#comment-16356556
]
Hive QA commented on HIVE-18238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909596/HIVE-18238.08.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9086/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9086/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9086/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-08 06:55:44.878
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9086/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-08 06:55:44.881
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 16b8575 HIVE-18636 - fix TestTxnNoBuckets.testCTAS - keeps
failing on ptest (Eugene Koifman, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 16b8575 HIVE-18636 - fix TestTxnNoBuckets.testCTAS - keeps
failing on ptest (Eugene Koifman, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-08 06:55:47.335
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9086/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:2435
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7122
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' with
conflicts.
Going to apply patch with: git apply -p0
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:2435
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7122
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909596 - PreCommit-HIVE-Build
> Driver execution may not have configuration changing sideeffects
> -
>
> Key: HIVE-18238
> URL: https://issues.apache.org/jira/browse/HIVE-18238
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18238.01wip01.patch, HIVE-18238.02.patch,
> HIVE-18238.03.patch, HIVE-18238.04.patch, HIVE-18238.04wip01.patch,
> HIVE-18238.07.patch, HIVE-18238.08.patch
>
>
> {{Driver}} executes sql statements which use "hiveconf" settings;
> but the {{Driver}} itself may *not* change the configuration...
> I've found an example; which shows how hazardous this is...
> {code}
> set hive.mapred.mode=strict;
> select "${hiveconf:hive.mapred.mode}";
> create table t (a int);
> analyze table t compute statistic
[jira] [Commented] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[
https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356553#comment-16356553
]
Hive QA commented on HIVE-18645:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909593/HIVE-18645.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=301)
org.apache.hadoop.hive.metastore.TestAcidTableSetup.testTransactionalValidation
(batchId=223)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients
(batchId=200)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9085/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9085/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9085/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909593 - PreCommit-HIVE-Build
> invalid url address in README.txt from module hbase-handler
> ---
>
> Key: HIVE-18645
> URL: https://issues.apache.org/jira/browse/HIVE-18645
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Trivial
> Attachments: HIVE-18645.1.patch
>
>
> The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in
> README.txt from module hbase-handler.
> Update the url and change .txt to .md
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18526) Backport HIVE-16886 to Hive 2
[ https://issues.apache.org/jira/browse/HIVE-18526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356530#comment-16356530 ] anishek commented on HIVE-18526: [~akolb] i think add this to branch-3 on top of the existing fix. If you want to do this as a different bug thats fine as well. reverting + reapplying a different one with huge time gap for revert does not make sense. > Backport HIVE-16886 to Hive 2 > - > > Key: HIVE-18526 > URL: https://issues.apache.org/jira/browse/HIVE-18526 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 2.3.3 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-18526.01-branch-2.patch, > HIVE-18526.02-branch-2.patch > > > The fix for HIVE-16886 isn't in Hive 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Attachment: HIVE-18645.1.patch > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Status: Patch Available (was: Open) > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Attachment: (was: HIVE-18645.1.patch) > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Status: Open (was: Patch Available) > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Attachment: (was: HIVE-18645.1.patch) > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[ https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18645: Attachment: HIVE-18645.1.patch > invalid url address in README.txt from module hbase-handler > --- > > Key: HIVE-18645 > URL: https://issues.apache.org/jira/browse/HIVE-18645 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Attachments: HIVE-18645.1.patch, HIVE-18645.1.patch > > > The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in > README.txt from module hbase-handler. > Update the url and change .txt to .md -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18645) invalid url address in README.txt from module hbase-handler
[
https://issues.apache.org/jira/browse/HIVE-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356511#comment-16356511
]
Hive QA commented on HIVE-18645:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 16b8575 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9085/yetus/whitespace-eol.txt
|
| modules | C: hbase-handler U: hbase-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9085/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> invalid url address in README.txt from module hbase-handler
> ---
>
> Key: HIVE-18645
> URL: https://issues.apache.org/jira/browse/HIVE-18645
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Saijin Huang
>Assignee: Saijin Huang
>Priority: Trivial
> Attachments: HIVE-18645.1.patch
>
>
> The url "http://wiki.apache.org/hadoop/Hive/HBaseIntegration"; is invalid in
> README.txt from module hbase-handler.
> Update the url and change .txt to .md
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356510#comment-16356510
]
Hive QA commented on HIVE-18622:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909578/HIVE-18622.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 204 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_date_1]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_1]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_nested_udf]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_casts]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=20)
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=244)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_date_1]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_number_compare_projection]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_ptf_part_simple]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_nested_udf]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_casts]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=156)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_nested_udf]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_math_funcs]
(batchId=115)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorLogicalExpressions.testBooleanNot
(batchId=283)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorLogicalExpressions.testLongColAndLongCol
(batchId=283)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorLogicalExpressions.testLongColOrLongCol
(batchId=283)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorScalarColArithmetic.testLongScalarSubtractLongColWithRepeating
(batchId=283)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testColConcatCol
(batchId=284)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testStringColCompareStringColProjection
(batchId=284)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testSubstrStart
(batchId=284)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorStringExpressions.testSubstrStartLen
(batchId=284)
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.TestColumnScalarOperationVectorExpressionEvaluation.testDoubleColAddDoubleScalar
[jira] [Commented] (HIVE-18622) Vectorization: IF Statements, Comparisons, and more do not handle NULLs correctly
[
https://issues.apache.org/jira/browse/HIVE-18622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356486#comment-16356486
]
Hive QA commented on HIVE-18622:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 1 new + 124 unchanged
- 4 fixed = 125 total (was 128) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 2 new + 1701 unchanged - 9
fixed = 1703 total (was 1710) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9084/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9084/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9084/yetus/whitespace-eol.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9084/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: IF Statements, Comparisons, and more do not handle NULLs
> correctly
> -
>
> Key: HIVE-18622
> URL: https://issues.apache.org/jira/browse/HIVE-18622
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HIVE-18622.03.patch
>
>
>
> Many vector expression classes are missing guards around setting noNulls
> among other things.
> {code:java}
> // Carefully update noNulls...
> if (outputColVector.noNulls) {
> outputColVector.noNulls = inputColVector.noNulls;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Resolution: Fixed Status: Resolved (was: Patch Available) committed to master thanks Sergey for the review > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[
https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356474#comment-16356474
]
Hive QA commented on HIVE-18626:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909706/HIVE-18626.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9083/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9083/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9083/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909706 - PreCommit-HIVE-Build
> Repl load "with" clause does not pass config to tasks
> -
>
> Key: HIVE-18626
> URL: https://issues.apache.org/jira/browse/HIVE-18626
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch,
> HIVE-18626.3.patch
>
>
> The "with" clause in repl load suppose to pass custom hive config entries to
> replication. However, the config is only effective in
> BootstrapEventsIterator, but not the generated tasks (such as MoveTask,
> DDLTask).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356455#comment-16356455
]
Aihua Xu commented on HIVE-18421:
-
[~vihangk1] Sorry for the late reply. I left comment in RB. Basically I don't
follow why we need both CHECKED and UNCHECKED implementations. Seems we should
only have CHECKED one if UNCHECKED one would generate incorrect result. The
user would get incorrect result without notice, right?
Of course, even we want to support UNCHECKED implementation, we should error
out/fail the query if there is overflow so the user knows to set the flag to
true. BTW: how much performance impact for this and why (don't exactly follow
previous discussion)?
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356435#comment-16356435 ] Jesus Camacho Rodriguez commented on HIVE-18359: +1 pending tests > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.10.patch, > HIVE-18359.11.patch, HIVE-18359.2.patch, HIVE-18359.3.patch, > HIVE-18359.4.patch, HIVE-18359.5.patch, HIVE-18359.6.patch, > HIVE-18359.7.patch, HIVE-18359.8.patch, HIVE-18359.9.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[
https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356430#comment-16356430
]
Hive QA commented on HIVE-18626:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 171 unchanged - 4
fixed = 171 total (was 175) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9083/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl load "with" clause does not pass config to tasks
> -
>
> Key: HIVE-18626
> URL: https://issues.apache.org/jira/browse/HIVE-18626
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch,
> HIVE-18626.3.patch
>
>
> The "with" clause in repl load suppose to pass custom hive config entries to
> replication. However, the config is only effective in
> BootstrapEventsIterator, but not the generated tasks (such as MoveTask,
> DDLTask).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356426#comment-16356426
]
Ke Jia commented on HIVE-18340:
---
[~stakiar]：
{quote}Hive-on-Tez's has an implementation of DynamicValueRegistry that uses
some special Tez APIs such as ProcessorContext#waitForAllInputsReady, how are
we simulating this in HoS?
{quote}
[~kellyzly],Yes, For HoS, I flush the runtime filter info (min/max and bloom
filter) to hdfs in SparkRuntimeFilterPruningSinkOperator operator and get the
info from hdfs in SparkRuntimeFilterPruner , which is similar as
SparkPartitionPruningSinkOperator and SparkDynamicPartitionPruner class in
Spark DPP.
{quote}It would be nice to have some qtests to help visualize what the explain
plan with RF would look like
{quote}
I upload the HIVE-18340.2.patch to add qtest "spark_runtime_filter_pruning.q"
and "spark_runtime_filter_pruning.q.out".
Thanks [~stakiar], [~kellyzly] for your review!
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
>
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Ke Jia
>Assignee: Ke Jia
>Priority: Major
> Attachments: HIVE-18340.1.patch, HIVE-18340.2.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[ https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-18340: -- Attachment: HIVE-18340.2.patch > Dynamic Min-Max/BloomFilter runtime-filtering in HoS > > > Key: HIVE-18340 > URL: https://issues.apache.org/jira/browse/HIVE-18340 > Project: Hive > Issue Type: New Feature > Components: Spark >Affects Versions: 3.0.0 >Reporter: Ke Jia >Assignee: Ke Jia >Priority: Major > Attachments: HIVE-18340.1.patch, HIVE-18340.2.patch > > > Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269 > and we should implement the same in HOS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356423#comment-16356423
]
Prasanth Jayachandran commented on HIVE-18547:
--
I cannot repro this issue locally since this is a race condition. This patch
updates the cluster fraction of previously recorded events when the cluster
fraction changes.
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Status: Patch Available (was: Open)
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Affects Version/s: 3.0.0
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-18547:
Assignee: Prasanth Jayachandran
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18547) WM: trigger test may fail
[
https://issues.apache.org/jira/browse/HIVE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-18547:
-
Attachment: HIVE-18547.1.patch
> WM: trigger test may fail
> -
>
> Key: HIVE-18547
> URL: https://issues.apache.org/jira/browse/HIVE-18547
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-18547.1.patch
>
>
> https://builds.apache.org/job/PreCommit-HIVE-Build/8818/testReport/org.apache.hive.jdbc/TestTriggersMoveWorkloadManager/testTriggerMoveAndKill/
> Looks like the cluster allocation assignment and WM event creation race,
> probably because WM returns session to the caller ASAP and then makes the
> changes after that.
> {noformat}
> 'Event: GET Pool: BI Cluster %: 80.00' expected in STDERR capture, but not
> found.
> ...
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Processing changes for pool BI: [BI, query parallelism
> 1, fraction of the cluster 0.80011920929, fraction used by child pools
> 0.0, active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Starting 1 queries in pool [BI, query parallelism 1,
> fraction of the cluster 0.80011920929, fraction used by child pools 0.0,
> active sessions 0, initializing sessions 0]
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.WorkloadManager: Received a session from AM pool
> sessionId=2be29c62-9f2c-40b7-a5eb-6298baf83a34, queueName=default,
> user=hiveptest, doAs=false, isOpen=true, isDefault=true, expires in
> 588529859ms, WM state poolName=null, clusterFraction=0.0, queryId=null,
> killReason=null
> 2018-01-24T15:07:31,746 INFO [HiveServer2-Background-Pool: Thread-1377]
> tez.WmEvent: Added WMEvent: EventType: GET EventStartTimestamp: 1516835251746
> elapsedTime: 0 wmTezSessionInfo:SessionId:
> 2be29c62-9f2c-40b7-a5eb-6298baf83a34 Pool: BI Cluster %: 0.0
> 2018-01-24T15:07:31,746 INFO [Workload management master]
> tez.GuaranteedTasksAllocator: Updating 2be29c62-9f2c-40b7-a5eb-6298baf83a34
> with 3 guaranteed tasks
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[
https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356420#comment-16356420
]
Hive QA commented on HIVE-18636:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909556/HIVE-18636.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9082/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9082/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9082/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909556 - PreCommit-HIVE-Build
> fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
> --
>
> Key: HIVE-18636
> URL: https://issues.apache.org/jira/browse/HIVE-18636
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18636.01.patch
>
>
> need to update expected result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356410#comment-16356410
]
Sahil Takiar commented on HIVE-18034:
-
Actually looks like Spark's notion of totalDuration is how long a Spark task
takes to run (exposed by {{TaskInfo#duration()}}).
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18654) Add Hiveserver2 specific HADOOP_OPTS environment variable
[ https://issues.apache.org/jira/browse/HIVE-18654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-18654: -- > Add Hiveserver2 specific HADOOP_OPTS environment variable > -- > > Key: HIVE-18654 > URL: https://issues.apache.org/jira/browse/HIVE-18654 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Minor > > HIVE-2665 added support to include metastore specific HADOOP_OPTS variable. > This is helpful in debugging especially if you want to add some jvm > parameters to metastore's process. A similar setting for Hiveserver2 is > missing and could be very helpful in debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: (was: HIVE-18653.1.patch) > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356403#comment-16356403 ] Prasanth Jayachandran commented on HIVE-18653: -- [~djaiswal] can you plz take a look? > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: HIVE-18653.1.patch > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Status: Patch Available (was: Open) > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18653: - Attachment: HIVE-18653.1.patch > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18653.1.patch > > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18653) Fix TestOperators test failure in master
[ https://issues.apache.org/jira/browse/HIVE-18653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-18653: > Fix TestOperators test failure in master > > > Key: HIVE-18653 > URL: https://issues.apache.org/jira/browse/HIVE-18653 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > HIVE-17848 is causing TestOperators#testNoConditionalTaskSizeForLlap to fail > in master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356394#comment-16356394
]
Vihang Karajgaonkar commented on HIVE-14171:
cc [~Ferd]
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356393#comment-16356393
]
Sahil Takiar edited comment on HIVE-18034 at 2/8/18 2:33 AM:
-
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime + executorDeserializeTime}} should give
us the total executor duration, and {{jvmGCTime}} should give the GC time. I
think a {{LOG#warn}} and a warning on the console should be printed.
was (Author: stakiar):
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime}} + {{ executorDeserializeTime}} should
give us the total executor duration, and {{jvmGCTime}} should give the GC time.
I think a {{LOG#warn}} and a warning on the console should be printed.
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356392#comment-16356392
]
Vihang Karajgaonkar commented on HIVE-14171:
AFAIK {{hive.vectorized.use.row.serde.deserialize}} is not supported for
Parquet. Does it fail if you set to false as well? Many fixes for Parquet
vectorized reader went in 2.3.0. So it would be good to check using 2.3.0 if
its possible.
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
> ... 17 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC
[
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356393#comment-16356393
]
Sahil Takiar commented on HIVE-18034:
-
Looks like the Spark Web UI uses the following logic to detect if excessive GC
time has been spent:
{code}
// totalDuration range from 0 to 50% GC time, alpha max = 1
function totalDurationAlpha(totalGCTime, totalDuration) {
return totalDuration > 0 ?
(Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1;
}
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
var GCTimePercent = 0.1;
function totalDurationStyle(totalGCTime, totalDuration) {
// Red if GC time over GCTimePercent of total time
return (totalGCTime > GCTimePercent * totalDuration) ?
("hsla(0, 100%, 50%, " + totalDurationAlpha(totalGCTime, totalDuration)
+ ")") : "";
{code}
We should be able to use the info in {{o.a.hive.spark.client.metrics}} to do
the same calculation. {{executorRunTime}} + {{ executorDeserializeTime}} should
give us the total executor duration, and {{jvmGCTime}} should give the GC time.
I think a {{LOG#warn}} and a warning on the console should be printed.
> Improving logging with HoS executors spend lots of time in GC
> -
>
> Key: HIVE-18034
> URL: https://issues.apache.org/jira/browse/HIVE-18034
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> There are times when Spark will spend lots of time doing GC. The Spark
> History UI shows a bunch of red flags when too much time is spent in GC. It
> would be nice if those warnings are propagated to Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[
https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356369#comment-16356369
]
Hive QA commented on HIVE-18636:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 18 new + 261 unchanged - 18
fixed = 279 total (was 279) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9082/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9082/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
> --
>
> Key: HIVE-18636
> URL: https://issues.apache.org/jira/browse/HIVE-18636
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-18636.01.patch
>
>
> need to update expected result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18340) Dynamic Min-Max/BloomFilter runtime-filtering in HoS
[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356365#comment-16356365
]
liyunzhang commented on HIVE-18340:
---
[~stakiar]: {quote}
Hive-on-Tez's has an implementation of DynamicValueRegistry that uses some
special Tez APIs such as ProcessorContext#waitForAllInputsReady, how are we
simulating this in HoS?
{quote}
ProcessorContext#waitForAllInputsReady is called by
{{org.apache.hadoop.hive.ql.exec.tez.DynamicValueRegistryTez#init}} to read the
runtime filter info. For HoS, I guess [~Jk_self] will read the info from hdfs
which is similar as Spark DPP.
If my understanding is not right, [~stakiar], [~Jk_Self] please tell me.
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
>
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
>Affects Versions: 3.0.0
>Reporter: Ke Jia
>Assignee: Ke Jia
>Priority: Major
> Attachments: HIVE-18340.1.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18578) Some class has missed the ASF header
[ https://issues.apache.org/jira/browse/HIVE-18578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356362#comment-16356362 ] Saijin Huang commented on HIVE-18578: - [~kgyrtkirk] thank you for your commit! > Some class has missed the ASF header > > > Key: HIVE-18578 > URL: https://issues.apache.org/jira/browse/HIVE-18578 > Project: Hive > Issue Type: Bug >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-18578.1.patch > > > Some class has missed the ASF header -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[
https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356353#comment-16356353
]
Hive QA commented on HIVE-15911:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853014/0001-Hive-15911.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12970 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_view_as_select_view]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9081/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9081/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9081/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853014 - PreCommit-HIVE-Build
> Create a view based on another view throws an exception “FAILED:
> NullPointerException null”
> ---
>
> Key: HIVE-15911
> URL: https://issues.apache.org/jira/browse/HIVE-15911
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.1.1
> Environment: hive2.1.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: 0001-Hive-15911.patch
>
>
> when I create a new view based on another view I an exception “FAILED:
> NullPointerException null”:
> hive> create view view2(a,b) as select a, b from view1; //view1 is another
> view
> FAILED: NullPointerException null
> hive>
> The hive log show error stack：
> 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116))
> - FAILED: NullPointerException null
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1239)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1229)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.a
[jira] [Commented] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356343#comment-16356343 ] Ashutosh Chauhan commented on HIVE-18607: - +1 > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[ https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356325#comment-16356325 ] KaiXu edited comment on HIVE-14171 at 2/8/18 1:17 AM: -- Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format. environment: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 stack trace: Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30) at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) ... 21 more was (Author: kaixu): Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apa
[jira] [Updated] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-18626: -- Attachment: HIVE-18626.3.patch > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch, > HIVE-18626.3.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356328#comment-16356328 ] Daniel Dai commented on HIVE-18626: --- Another case need to pass configuration to task. > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch, HIVE-18626.2.patch, > HIVE-18626.3.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[ https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356325#comment-16356325 ] KaiXu edited comment on HIVE-14171 at 2/8/18 1:16 AM: -- Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Hive2.2.0 with patch HIVE-14029 Spark2.0.2 Hadoop2.7.3 Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30) at org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83) at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47) at org.apache.spark.scheduler.Task.run(Task.scala:86) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228) ... 17 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51) at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) ... 21 more was (Author: kaixu): Found similar issue with hive.vectorized.use.row.serde.deserialize=true on TPC-DS query12, parquet file format: Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2): java.io.IOException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121) at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShim
[jira] [Commented] (HIVE-14171) Parquet: Simple vectorization throws NPEs
[
https://issues.apache.org/jira/browse/HIVE-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356325#comment-16356325
]
KaiXu commented on HIVE-14171:
--
Found similar issue with hive.vectorized.use.row.serde.deserialize=true on
TPC-DS query12, parquet file format:
Job aborted due to stage failure: Task 76 in stage 1.0 failed 4 times, most
recent failure: Lost task 76.3 in stage 1.0 (TID 35, skl-slave2):
java.io.IOException: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:231)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:141)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:254)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at
scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:83)
at
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:157)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:51)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:228)
... 17 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:51)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 21 more
> Parquet: Simple vectorization throws NPEs
> -
>
> Key: HIVE-14171
> URL: https://issues.apache.org/jira/browse/HIVE-14171
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Priority: Major
> Labels: Parquet
>
> {code}
> create temporary table cd_parquet stored as parquet as select * from
> customer_demographics;
> select count(1) from cd_parquet where cd_gender = 'F';
> {code}
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:206)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:118)
> at
> org.apache.hadoop.hive.q
[jira] [Commented] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356322#comment-16356322
]
Sahil Takiar commented on HIVE-17837:
-
Rebased patch. [~lirui] could you take a look? I checked the code and there
shouldn't be much additional overhead to calling
{{RemoteSparkJobStatus#isRemoteActive}} multiple times. The method essentially
boils down to checking a few boolean variables, and acquiring / releasing a
lock while checking the state of the underlying {{SocketChannel}}.
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17837.1.patch, HIVE-17837.2.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17837) Explicitly check if the HoS Remote Driver has been lost in the RemoteSparkJobMonitor
[
https://issues.apache.org/jira/browse/HIVE-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17837:
Attachment: HIVE-17837.2.patch
> Explicitly check if the HoS Remote Driver has been lost in the
> RemoteSparkJobMonitor
> -
>
> Key: HIVE-17837
> URL: https://issues.apache.org/jira/browse/HIVE-17837
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-17837.1.patch, HIVE-17837.2.patch
>
>
> Right now the {{RemoteSparkJobMonitor}} implicitly checks if the connection
> to the Spark remote driver is active. It does this everytime it triggers an
> invocation of the {{Rpc#call}} method (so any call to {{SparkClient#run}}).
> There are scenarios where we have seen the {{RemoteSparkJobMonitor}} when the
> connection to the driver dies, because the implicit call fails to be invoked
> (see HIVE-15860).
> It would be ideal if we made this call explicit, so we fail as soon as we
> know that the connection to the driver has died.
> The fix has the added benefit that it allows us to fail faster in the case
> where the {{RemoteSparkJobMonitor}} is in the QUEUED / SENT state. If its
> stuck in that state, it won't fail until it hits the monitor timeout (by
> default 1 minute), even though we already know the connection has died. The
> error message that is thrown is also a little imprecise, it says there could
> be queue contention, even though we know the real reason is that the
> connection was lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18598) Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356299#comment-16356299 ] Ashutosh Chauhan commented on HIVE-18598: - +1 > Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch, HIVE-18598.2.patch, > HIVE-18598.3.patch > > > HIVE-16605 is enabling/enforcing NOT NULL constraint. But since Hive do not > manage the data for external tables and can not enforce constraints it > doesn't make sense to allow NOT NULL constraints to be enabled/enforced on > external table. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15911) Create a view based on another view throws an exception “FAILED: NullPointerException null”
[
https://issues.apache.org/jira/browse/HIVE-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356295#comment-16356295
]
Hive QA commented on HIVE-15911:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 429 unchanged - 0
fixed = 431 total (was 429) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9081/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create a view based on another view throws an exception “FAILED:
> NullPointerException null”
> ---
>
> Key: HIVE-15911
> URL: https://issues.apache.org/jira/browse/HIVE-15911
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.1.1
> Environment: hive2.1.0
>Reporter: yangfang
>Assignee: yangfang
>Priority: Major
> Attachments: 0001-Hive-15911.patch
>
>
> when I create a new view based on another view I an exception “FAILED:
> NullPointerException null”:
> hive> create view view2(a,b) as select a, b from view1; //view1 is another
> view
> FAILED: NullPointerException null
> hive>
> The hive log show error stack：
> 2017-02-15T15:40:25,816 ERROR ql.Driver (SessionState.java:printError(1116))
> - FAILED: NullPointerException null
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:863)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:552)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1319)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1459)
> at o
[jira] [Updated] (HIVE-18359) Extend grouping set limits from int to long
[ https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18359: - Attachment: HIVE-18359.11.patch > Extend grouping set limits from int to long > --- > > Key: HIVE-18359 > URL: https://issues.apache.org/jira/browse/HIVE-18359 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18359.1.patch, HIVE-18359.10.patch, > HIVE-18359.11.patch, HIVE-18359.2.patch, HIVE-18359.3.patch, > HIVE-18359.4.patch, HIVE-18359.5.patch, HIVE-18359.6.patch, > HIVE-18359.7.patch, HIVE-18359.8.patch, HIVE-18359.9.patch > > > Grouping sets is broken for >32 columns because of usage of Int for bitmap > (also GROUPING__ID virtual column). This assumption breaks grouping > sets/rollups/cube when number of participating aggregation columns is >32. > The easier fix would be extend it to Long for now. The correct fix would be > to use BitSets everywhere but that would require GROUPING__ID column type to > binary which will make predicates on GROUPING__ID difficult to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356273#comment-16356273 ] Ashutosh Chauhan commented on HIVE-18573: - Some of test failures seems related. > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, > HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch, > HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators, > this is to prepare the ground for pushing complex expressions to > Druid-calcite adapter. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356278#comment-16356278
]
Hive QA commented on HIVE-18629:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909544/HIVE-18629.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12969 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=235)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9080/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909544 - PreCommit-HIVE-Build
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18586) Upgrade Derby to 10.14.1.0
[ https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356274#comment-16356274 ] Aihua Xu commented on HIVE-18586: - [~alangates] Can you help take a look at the test change [~janulatha] is making? What is the original purpose to add SecurityManager to check System.exit()? Thanks. > Upgrade Derby to 10.14.1.0 > -- > > Key: HIVE-18586 > URL: https://issues.apache.org/jira/browse/HIVE-18586 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch, > HIVE-18586.3.patch, HIVE-18586.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: (was: HIVE-18625.2.patch)
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18625:
--
Attachment: HIVE-18625.2.patch
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch, HIVE-18625.2.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356235#comment-16356235
]
Sergey Shelukhin commented on HIVE-18629:
-
Updated to also handle isRepeating case.
[~mmccline] I don't think the changes around noNulls are necessary... they can
only become relevant if input vector is invalid in the first place and has
isNull elements when noNulls is set.
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18629:
Attachment: HIVE-18629.02.patch
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch,
> HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356232#comment-16356232
]
Hive QA commented on HIVE-18629:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} storage-api: The patch generated 1 new + 4 unchanged -
2 fixed = 5 total (was 6) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed
= 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-ql.txt
|
| modules | C: storage-api ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffe
[jira] [Updated] (HIVE-18598) Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18598: --- Status: Open (was: Patch Available) > Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch, HIVE-18598.2.patch, > HIVE-18598.3.patch > > > HIVE-16605 is enabling/enforcing NOT NULL constraint. But since Hive do not > manage the data for external tables and can not enforce constraints it > doesn't make sense to allow NOT NULL constraints to be enabled/enforced on > external table. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18598) Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18598: --- Status: Patch Available (was: Open) > Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch, HIVE-18598.2.patch, > HIVE-18598.3.patch > > > HIVE-16605 is enabling/enforcing NOT NULL constraint. But since Hive do not > manage the data for external tables and can not enforce constraints it > doesn't make sense to allow NOT NULL constraints to be enabled/enforced on > external table. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18598) Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18598: --- Attachment: HIVE-18598.3.patch > Disallow NOT NULL constraints to be ENABLED/ENFORCED with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch, HIVE-18598.2.patch, > HIVE-18598.3.patch > > > HIVE-16605 is enabling/enforcing NOT NULL constraint. But since Hive do not > manage the data for external tables and can not enforce constraints it > doesn't make sense to allow NOT NULL constraints to be enabled/enforced on > external table. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18650) Add compression related tests for blobstores
[ https://issues.apache.org/jira/browse/HIVE-18650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhou Sun updated HIVE-18650: -- Status: Patch Available (was: Open) > Add compression related tests for blobstores > > > Key: HIVE-18650 > URL: https://issues.apache.org/jira/browse/HIVE-18650 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.3.2 >Reporter: Yuzhou Sun >Assignee: Yuzhou Sun >Priority: Major > Attachments: HIVE-18650.patch, over10k.bz2, s_aa.gz, s_ab.gz, > s_ac.gz, s_ad.gz, s_ae.gz, s_af.gz, s_ag.gz, s_ah.gz, s_ai.gz, s_aj.gz > > > This patch introduces the following tests and files into the hive-blobstore > module: > * bzip_read.q -> Test read bzip compressed input data > * bzip_write.q -> Test write gzip compressed output table > * gzip_read.q -> Test read gzip compressed input data > * gzip_write.q -> Test write gzip compressed output table > * gzip_splits.q -> Test read splited gzip compressed input data > * over10k.bz2 -> Bzip compressed file of over10k > * gzip_splits/ -> 10 gzip files of 21-22 KB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18650) Add compression related tests for blobstores
[ https://issues.apache.org/jira/browse/HIVE-18650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhou Sun updated HIVE-18650: -- Affects Version/s: 2.3.2 > Add compression related tests for blobstores > > > Key: HIVE-18650 > URL: https://issues.apache.org/jira/browse/HIVE-18650 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.3.2 >Reporter: Yuzhou Sun >Assignee: Yuzhou Sun >Priority: Major > Attachments: HIVE-18650.patch, over10k.bz2, s_aa.gz, s_ab.gz, > s_ac.gz, s_ad.gz, s_ae.gz, s_af.gz, s_ag.gz, s_ah.gz, s_ai.gz, s_aj.gz > > > This patch introduces the following tests and files into the hive-blobstore > module: > * bzip_read.q -> Test read bzip compressed input data > * bzip_write.q -> Test write gzip compressed output table > * gzip_read.q -> Test read gzip compressed input data > * gzip_write.q -> Test write gzip compressed output table > * gzip_splits.q -> Test read splited gzip compressed input data > * over10k.bz2 -> Bzip compressed file of over10k > * gzip_splits/ -> 10 gzip files of 21-22 KB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18650) Add compression related tests for blobstores
[ https://issues.apache.org/jira/browse/HIVE-18650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhou Sun updated HIVE-18650: -- Attachment: HIVE-18650.patch > Add compression related tests for blobstores > > > Key: HIVE-18650 > URL: https://issues.apache.org/jira/browse/HIVE-18650 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Yuzhou Sun >Assignee: Yuzhou Sun >Priority: Major > Attachments: HIVE-18650.patch, over10k.bz2, s_aa.gz, s_ab.gz, > s_ac.gz, s_ad.gz, s_ae.gz, s_af.gz, s_ag.gz, s_ah.gz, s_ai.gz, s_aj.gz > > > This patch introduces the following tests and files into the hive-blobstore > module: > * bzip_read.q -> Test read bzip compressed input data > * bzip_write.q -> Test write gzip compressed output table > * gzip_read.q -> Test read gzip compressed input data > * gzip_write.q -> Test write gzip compressed output table > * gzip_splits.q -> Test read splited gzip compressed input data > * over10k.bz2 -> Bzip compressed file of over10k > * gzip_splits/ -> 10 gzip files of 21-22 KB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356205#comment-16356205
]
Hive QA commented on HIVE-18573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909533/HIVE-18573.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 47 failed/errored test(s), 12994 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ba_table_udfs]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_binary_data]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_length] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=59)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_varchar_1]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf1]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf_adaptor_1]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=133)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=231)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9077/testReport
Console output: https://builds.apache.o
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[
https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356214#comment-16356214
]
Vihang Karajgaonkar commented on HIVE-18629:
Patch looks good to me. Can we include a test case which would have caused the
NPE before the patch?
> copyValues in BytesColumnVector may be missing null checks
> --
>
> Key: HIVE-18629
> URL: https://issues.apache.org/jira/browse/HIVE-18629
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18629.01.patch, HIVE-18629.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133)
> {noformat}
> IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case
> has isNull checks for v2/v3 buffers that copySelected is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18627: --- Attachment: HIVE-18627.3.patch > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch, HIVE-18627.2.patch, > HIVE-18627.3.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18627: --- Attachment: (was: HIVE-18627.3.patch) > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch, HIVE-18627.2.patch, > HIVE-18627.3.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18627) PPD: Handle FLOAT boxing differently for single/double precision constants
[ https://issues.apache.org/jira/browse/HIVE-18627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-18627: --- Attachment: HIVE-18627.3.patch > PPD: Handle FLOAT boxing differently for single/double precision constants > -- > > Key: HIVE-18627 > URL: https://issues.apache.org/jira/browse/HIVE-18627 > Project: Hive > Issue Type: Bug > Components: Types >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-18627.1.patch, HIVE-18627.2.patch, > HIVE-18627.3.patch > > > Constants like 0.1 and 0.3 are differently boxed based on intermediate > precision of the compiler codepath. > Disabling CBO produces 0.1BD constants which fail to box correctly to > Double/Float. > Enabling CBO fixes this issue, but cannot be applied all queries in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18650) Add compression related tests for blobstores
[ https://issues.apache.org/jira/browse/HIVE-18650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuzhou Sun reassigned HIVE-18650: - > Add compression related tests for blobstores > > > Key: HIVE-18650 > URL: https://issues.apache.org/jira/browse/HIVE-18650 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Yuzhou Sun >Assignee: Yuzhou Sun >Priority: Major > Attachments: over10k.bz2, s_aa.gz, s_ab.gz, s_ac.gz, s_ad.gz, > s_ae.gz, s_af.gz, s_ag.gz, s_ah.gz, s_ai.gz, s_aj.gz > > > This patch introduces the following tests and files into the hive-blobstore > module: > * bzip_read.q -> Test read bzip compressed input data > * bzip_write.q -> Test write gzip compressed output table > * gzip_read.q -> Test read gzip compressed input data > * gzip_write.q -> Test write gzip compressed output table > * gzip_splits.q -> Test read splited gzip compressed input data > * over10k.bz2 -> Bzip compressed file of over10k > * gzip_splits/ -> 10 gzip files of 21-22 KB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356213#comment-16356213
]
Vihang Karajgaonkar commented on HIVE-18421:
[~mmccline] would you be able to take a look at this? This patch introduces new
checked vector expressions and uses them when we set the newly introduced
config {{hive.vectorized.use.checked.expressions}}. I introduced checked
expressions for the arithmetic operators and some others where I could see the
issue could cause different results after overflow based on my analysis.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18607: Attachment: HIVE-18607.04.patch > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356212#comment-16356212 ] Sergey Shelukhin commented on HIVE-18607: - [~ashutoshc] can you take a look? not sure if anyone is familiar with HBase handler anymore. > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.04.patch, HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[
https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356210#comment-16356210
]
Hive QA commented on HIVE-17983:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12909534/HIVE-17983.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9078/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9078/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9078/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-07 23:04:24.529
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9078/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-07 23:04:24.532
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at ee2c814 HIVE-18642 : incorrect assertion in TezSessionPool for
WM (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at ee2c814 HIVE-18642 : incorrect assertion in TezSessionPool for
WM (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-07 23:04:27.095
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9078/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql:1
error: metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql: patch
does not apply
error: patch failed:
metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql:1
error: metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql: patch
does not apply
error: patch failed:
metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql:1
error: metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql:
patch does not apply
error: patch failed:
metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql:615
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql' with
conflicts.
fatal: git apply: bad git-diff - inconsistent old filename on line 6283
fatal: git diff header lacks filename information when removing 2 leading
pathname components (line 39)
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12909534 - PreCommit-HIVE-Build
> Make the standalone metastore generate tarballs etc.
>
>
> Key: HIVE-17983
> URL: https://issues.apache.org/jira/browse/HIVE-17983
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17983.2.patch, HIVE-17983.3.patch,
> HIVE-17983.4.patch, HIVE-17983.patch
>
>
> In order to be separately installable the standalone metastore needs its own
> tarballs, startup scripts, etc. All of the SQL installation and upgrade
> scripts also need to move from metastore to standalone-metastore.
> I also plan to create Dockerfiles for different database types so that
> developers can test the SQL installation a
[jira] [Updated] (HIVE-18607) HBase HFile write does strange things
[ https://issues.apache.org/jira/browse/HIVE-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18607: Attachment: (was: HIVE-18607.03.patch) > HBase HFile write does strange things > - > > Key: HIVE-18607 > URL: https://issues.apache.org/jira/browse/HIVE-18607 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18607.02.patch, HIVE-18607.03.patch, > HIVE-18607.patch > > > There's some strange code in the output handler that changes output directory > into a file because Hive supposedly wants that. > If you run insert overwrite with a side directory multiple times, the 2nd > insert fails -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356202#comment-16356202
]
Vihang Karajgaonkar commented on HIVE-18421:
Okay, I finally figured out the problem with vectorization_numeric_overflows.q.
Turns out on CentOS the ordering of rows is different for a query result when
compared to Ubuntu/MacOS. Both the results are valid since the key on which the
order by clause was issued have same value on 2 rows. So the order of these 2
rows with the same value of key is non-deterministic. Fixed the query so that
it orders by a different column which is always deterministic.
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356186#comment-16356186 ] Sergey Shelukhin commented on HIVE-18350: - +1 from my side, pending others' feedback, and tests > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.10.patch, > HIVE-18350.11.patch, HIVE-18350.12.patch, HIVE-18350.13.patch, > HIVE-18350.14.patch, HIVE-18350.15.patch, HIVE-18350.16.patch, > HIVE-18350.2.patch, HIVE-18350.3.patch, HIVE-18350.4.patch, > HIVE-18350.5.patch, HIVE-18350.6.patch, HIVE-18350.7.patch, > HIVE-18350.8.patch, HIVE-18350.9.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18421) Vectorized execution handles overflows in a different manner than non-vectorized execution
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-18421:
---
Attachment: HIVE-18421.07.patch
> Vectorized execution handles overflows in a different manner than
> non-vectorized execution
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18421.01.patch, HIVE-18421.02.patch,
> HIVE-18421.03.patch, HIVE-18421.04.patch, HIVE-18421.05.patch,
> HIVE-18421.06.patch, HIVE-18421.07.patch
>
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: HIVE-18614.patch > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Patch Available (was: Open) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Status: Open (was: Patch Available) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18614) Fix sys db creation in Hive
[ https://issues.apache.org/jira/browse/HIVE-18614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-18614: -- Attachment: (was: HIVE-18614.patch) > Fix sys db creation in Hive > --- > > Key: HIVE-18614 > URL: https://issues.apache.org/jira/browse/HIVE-18614 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18614.patch > > > Sys db can not be created due to several server side issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18575: Attachment: HIVE-18575.02.patch > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.01.patch, HIVE-18575.02.patch, > HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18649: -- Component/s: Transactions > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There also exists a JobConf field "job" in HIF; and a separate JobConf input > argument to some methods. These methods apply some changes to one jobconf and > some to another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18649: Description: Follow-up from HIVE-18575 ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There also exists a JobConf field "job" in HIF; and a separate JobConf input argument to some methods. These methods apply some changes to one jobconf and some to another, for no clear reason. was: Follow-up from HIVE-18575 ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There is also a job field with JobConf, and an input to some methods that is also JobConf; some methods apply some changes to one jobconf and some to another, for no clear reason. > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There also exists a JobConf field "job" in HIF; and a separate JobConf input > argument to some methods. These methods apply some changes to one jobconf and > some to another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[ https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356164#comment-16356164 ] Thejas M Nair commented on HIVE-17983: -- +1 pending tests Thanks for adding the docker support for being able to test scripts changes easily! That really helps with developer productivity! It would be good to get this in soon, as there are some other db script changes in pipeline blocked on this (ie, they would conflict with this) > Make the standalone metastore generate tarballs etc. > > > Key: HIVE-17983 > URL: https://issues.apache.org/jira/browse/HIVE-17983 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17983.2.patch, HIVE-17983.3.patch, > HIVE-17983.4.patch, HIVE-17983.patch > > > In order to be separately installable the standalone metastore needs its own > tarballs, startup scripts, etc. All of the SQL installation and upgrade > scripts also need to move from metastore to standalone-metastore. > I also plan to create Dockerfiles for different database types so that > developers can test the SQL installation and upgrade scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17875) Vectorization support for complex types breaks parquet vectorization
[
https://issues.apache.org/jira/browse/HIVE-17875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-17875:
--
Assignee: (was: Vihang Karajgaonkar)
> Vectorization support for complex types breaks parquet vectorization
>
>
> Key: HIVE-17875
> URL: https://issues.apache.org/jira/browse/HIVE-17875
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> HIVE-16589 introduced support for complex types for vectorized execution. It
> introduces two new configs {{hive.vectorized.complex.types.enabled}} and
> {{hive.vectorized.groupby.complex.types.enabled}} which default to true and
> control whether {{Vectorizer}} creates a vectorized execution plan for
> queries using complex types. Since Parquet fileformat does not support
> vectorization for complex types yet, any query running on parquet tables with
> complex types current fails with a RuntimeException complaining that the
> complex type is not supported. We should improve the logic in Vectorizer to
> check if the FileinputFormat supports complex types and if not it should not
> vectorize the query plan.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17397) vector_outer_join4.q.out explain plan not formatted correctly
[
https://issues.apache.org/jira/browse/HIVE-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-17397:
--
Assignee: (was: Vihang Karajgaonkar)
> vector_outer_join4.q.out explain plan not formatted correctly
> -
>
> Key: HIVE-17397
> URL: https://issues.apache.org/jira/browse/HIVE-17397
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Priority: Trivial
>
> {{vector_outer_join4.q}} user {{explain vectorization detail formatted}}
> which just dumps a JSON string without any indentations and new lines.
> Either, there is no option for {{explain vectorization detail formatted}} and
> it should just be {{explain vectorization detail}} or this may be a bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356149#comment-16356149 ] Sergey Shelukhin commented on HIVE-18649: - cc [~ekoifman] > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There is also a job field with JobConf, and an input to some methods that is > also JobConf; some methods apply some changes to one jobconf and some to > another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18649: Description: Follow-up from HIVE-11985 ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There is also a job field with JobConf, and an input to some methods that is also JobConf; some methods apply some changes to one jobconf and some to another, for no clear reason. was:ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There is also a job field with JobConf, and an input to some methods that is also JobConf; some methods apply some changes to one jobconf and some to another, for no clear reason. > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-11985 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There is also a job field with JobConf, and an input to some methods that is > also JobConf; some methods apply some changes to one jobconf and some to > another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18649: Description: Follow-up from HIVE-18575 ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There is also a job field with JobConf, and an input to some methods that is also JobConf; some methods apply some changes to one jobconf and some to another, for no clear reason. was: Follow-up from HIVE-11985 ACID properties may be added to the same job object for multiple tables, at least by the looks of it. There is also a job field with JobConf, and an input to some methods that is also JobConf; some methods apply some changes to one jobconf and some to another, for no clear reason. > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There is also a job field with JobConf, and an input to some methods that is > also JobConf; some methods apply some changes to one jobconf and some to > another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356139#comment-16356139
]
Hive QA commented on HIVE-18573:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 6 new + 60 unchanged - 0 fixed
= 66 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / ee2c814 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9077/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9077/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch,
> HIVE-18573.4.patch, HIVE-18573.5.patch, HIVE-18573.6.patch,
> HIVE-18573.7.patch, HIVE-18573.7.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during
> Query planning. It will be more beneficial to use proper calcite operators,
> this is to prepare the ground for pushing complex expressions to
> Druid-calcite adapter.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18456) Add some tests for HIVE-18367 to check that the table information contains the query correctly
[ https://issues.apache.org/jira/browse/HIVE-18456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18456: -- Attachment: HIVE-18456.2.patch > Add some tests for HIVE-18367 to check that the table information contains > the query correctly > -- > > Key: HIVE-18456 > URL: https://issues.apache.org/jira/browse/HIVE-18456 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18456.1.patch, HIVE-18456.2.patch > > > This cannot be tested with a CliDriver test so add a java test to check the > output of 'describe extended', which is changed by HIVE-18367 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18625) SessionState Not Checking For Directory Creation Result
[
https://issues.apache.org/jira/browse/HIVE-18625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356130#comment-16356130
]
Sahil Takiar commented on HIVE-18625:
-
Few comments
{code}
boolean created = fs.mkdirs(path, fsPermission);
if (!created) {
throw new IOException("mkdirs failed to create " + path + " on fs " + fs);
}
{code}
Can be simplified to:
{code}
if (!fs.mkdirs(...) { ... }
{code}
Think the exception message should be something like - "Failed to create
directory ...". What does {{fs#toString()}} print out?
You probably want to split {{testCreatePath}} into multiple tests.
> SessionState Not Checking For Directory Creation Result
> ---
>
> Key: HIVE-18625
> URL: https://issues.apache.org/jira/browse/HIVE-18625
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0, 2.3.2
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Minor
> Attachments: HIVE-18625.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L773
> {code:java}
> private static void createPath(HiveConf conf, Path path, String permission,
> boolean isLocal,
> boolean isCleanUp) throws IOException {
> FsPermission fsPermission = new FsPermission(permission);
> FileSystem fs;
> if (isLocal) {
> fs = FileSystem.getLocal(conf);
> } else {
> fs = path.getFileSystem(conf);
> }
> if (!fs.exists(path)) {
> fs.mkdirs(path, fsPermission);
> String dirType = isLocal ? "local" : "HDFS";
> LOG.info("Created " + dirType + " directory: " + path.toString());
> }
> if (isCleanUp) {
> fs.deleteOnExit(path);
> }
> }
> {code}
> The method {{fs.mkdirs(path, fsPermission)}} returns a boolean value
> indicating if the directory creation was successful or not. Hive ignores
> this return value and therefore could be acting on a directory that doesn't
> exist.
> Please capture the result, check it, and throw an Exception if it failed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18513) Query results caching
[ https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-18513: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) I've committed this to master. Thanks for review [~jcamachorodriguez] > Query results caching > - > > Key: HIVE-18513 > URL: https://issues.apache.org/jira/browse/HIVE-18513 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch, > HIVE-18513.3.patch, HIVE-18513.4.patch, HIVE-18513.5.patch, HIVE-18513.6.patch > > > Add a query results cache that can save the results of an executed Hive query > for reuse on subsequent queries. This may be useful in cases where the same > query is issued many times, since Hive can return back the results of a > cached query rather than having to execute the full query on the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
