[jira] [Updated] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22201:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Himanshu!
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[
https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934064#comment-16934064
]
Hive QA commented on HIVE-22205:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
33s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} llap-client in master has 26 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} itests/util in master has 51 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch llap-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 2
fixed = 6 total (was 8) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} The patch llap-server passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch kafka-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934063#comment-16934063
]
Hive QA commented on HIVE-22084:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 6m
13s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18660/dev-support/hive-personality.sh
|
| git revision | master / 5d1a949 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18660/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch,
> HIVE-22084.03.patch, HIVE-22084.04.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[
https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934040#comment-16934040
]
Hive QA commented on HIVE-22205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980766/HIVE-22205.3.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 16797 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=289)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=105)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=106)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=107)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=108)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=109)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=110)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver
(batchId=295)
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
(batchId=291)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriter
(batchId=358)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriterInputStream
(batchId=358)
org.apache.hive.streaming.TestStreaming.testAutoRollTransactionBatch
(batchId=358)
org.apache.hive.streaming.TestStreaming.testBucketingWhereBucketColIsNotFirstCol
(batchId=358)
org.apache.hive.streaming.TestStreaming.testCommitWithKeyValue (batchId=358)
org.apache.hive.streaming.TestStreaming.testConnectionWithWriteId (batchId=358)
org.apache.hive.streaming.TestStreaming.testInterleavedTransactionBatchCommits
(batchId=358)
org.apache.hive.streaming.TestStreaming.testMultipleTransactionBatchCommits
(batchId=358)
org.apache.hive.streaming.TestStreaming.testNoBuckets (batchId=358)
org.apache.hive.streaming.TestStreaming.testStreamBucketingMatchesRegularBucketing
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPStreamBucketingMatchesRegularBucketing
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevel
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevelMissingPartitionValues
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevelNonStringPartitionColumns
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testJsonInputStreamDP
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testRegexInputStreamDP
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteAfterAbort
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteAfterClose
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteBeforeBegin
(batchId=358)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18659/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18659/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18659/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980766 - PreCommit-HIVE-Build
> Upgrade zookeeper and curator versions
> --
>
> Key: HIVE-22205
> URL: https://issues.apache.org/jira/browse/HIVE-22205
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, HIVE-22205.patch
>
>
> Other components like hadoop have switched to using new ZK versions. So these
> jars end up in classpath for hive services and could cause issues due to
> in-compatible curator versions that hive uses.
> So it makes sense for hive to upgrade the ZK and curator versions to try to
> keep up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934021#comment-16934021
]
Himanshu Mishra commented on HIVE-22201:
[~ashutoshc] all tests passed now
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934016#comment-16934016 ] LuGuangMing commented on HIVE-22198: 【analyze】 explain SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; !image-2019-09-20-11-38-37-433.png|width=456,height=176!!image-2019-09-20-11-39-30-347.png|width=378,height=303! Analysis explian plan and resouce code, shows that stage-3 is added to the runnable queue when one of stage-2 or stage-6 runs is finished, due to ConditionalTask stage-9 and ConditionalTask stage-11 was invoked by parallel, remove the parent dependency of stage-3 on stage-2 and stage-6, The Driver scheduler finds that one of the stage-2 or stage-6 runtimes is finished, The children stage-3 will be found for scheduling add to runnable queue to run, At this time, there is still one node in stage-2 or stage-6 that has not run finished, and the result of the stage that has not finished will not be read by stage-3, so part of the result data is missing > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: image-2019-09-20-11-39-30-347.png > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Description: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment sql file create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; The result data should be two, but only one. was: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment sql file create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; -- result total number shoud be two result, but it's one create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: image-2019-09-20-11-38-37-433.png > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934003#comment-16934003
]
Hive QA commented on HIVE-21975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980763/HIVE-21975.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18658/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18658/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18658/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/sql/DriverManager.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/sql/SQLException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/jdbc/target/hive-jdbc-4.0.0-SNAPSHOT.jar(org/apache/hive/jdbc/HiveConnection.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/conf/Configured.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/NullWritable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/Text.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobClient.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobConf.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/Job.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/JobID.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/lib/output/NullOutputFormat.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/security/token/Token.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/Tool.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/thrift/libthrift/0.9.3-1/libthrift-0.9.3-1.jar(org/apache/thrift/TException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/conf/Configurable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Callable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/InterruptedException.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Boolean.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/ClassNotFoundException.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/ql/ErrorMsg.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Integer.class)]]
[loading
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933997#comment-16933997
]
Hive QA commented on HIVE-22209:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980761/HIVE-22209.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18657/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18657/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18657/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980761 - PreCommit-HIVE-Build
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Affects Version/s: 1.2.0 3.0.0 > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > -- result total number shoud be two result, but it's one > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22198 started by LuGuangMing. -- > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > -- result total number shoud be two result, but it's one > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Description: set parallel is true, set skewjoin is false, set auto convert join is false. run a unoin all, There is nothing error message, but some result data is missing, details check attatchment sql file create table tab1(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab2(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab3(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; create table tab4(tid int, com string) row format delimited fields terminated by '\t' stored as textfile; insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); set hive.auto.convert.join=false; set hive.optimize.skewjoin=true; set hive.exec.parallel=true; SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; -- result total number shoud be two result, but it's one create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; select * from test_parallel; was: it's missing a part of data. details check attatchment sql file > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > -- result total number shoud be two result, but it's one > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: test-parallel.sql > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel.sql > > > it's missing a part of data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Attachment: (was: test-parallel-skewJoin-unoinall.sql) > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel.sql > > > it's missing a part of data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933957#comment-16933957
]
Hive QA commented on HIVE-22209:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18657/dev-support/hive-personality.sh
|
| git revision | master / 5d1a949 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18657/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Description: it's missing a part of data. details check attatchment sql file was: connect hiveserver to set hive.auto.convert.join=false,hive.optimize.skewjoin=true, hive.exec.parallel=true, and run an unoin all with two child join query, result is wrong, it's missing data. details check attatchment sql file > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel-skewJoin-unoinall.sql > > > it's missing a part of data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Summary: Execute unoin-all with childs Join in parallel (was: parallel to run unoin-all missing a part of data) > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel-skewJoin-unoinall.sql > > > connect hiveserver to set > hive.auto.convert.join=false,hive.optimize.skewjoin=true, > hive.exec.parallel=true, and run an unoin all with two child join query, > result is wrong, it's missing data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22198) parallel to run unoin-all missing a part of data
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing updated HIVE-22198: --- Summary: parallel to run unoin-all missing a part of data (was: set parallel to run an unoin-all SQL and result missing a part of data) > parallel to run unoin-all missing a part of data > > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel-skewJoin-unoinall.sql > > > connect hiveserver to set > hive.auto.convert.join=false,hive.optimize.skewjoin=true, > hive.exec.parallel=true, and run an unoin all with two child join query, > result is wrong, it's missing data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22198) set parallel to run an unoin-all SQL and result missing a part of data
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LuGuangMing reassigned HIVE-22198: -- Assignee: LuGuangMing > set parallel to run an unoin-all SQL and result missing a part of data > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: test-parallel-skewJoin-unoinall.sql > > > connect hiveserver to set > hive.auto.convert.join=false,hive.optimize.skewjoin=true, > hive.exec.parallel=true, and run an unoin all with two child join query, > result is wrong, it's missing data. details check attatchment sql file > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933939#comment-16933939
]
Yuming Wang commented on HIVE-21237:
Yes. Hive 3.x has another issue HIVE-22097 that can't run on JDK 11.
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[
https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933928#comment-16933928
]
Hive QA commented on HIVE-20205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980759/HIVE-20205.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 16802 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=105)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=106)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=107)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=108)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=109)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.org.apache.hadoop.hive.cli.TestHBaseCliDriver
(batchId=110)
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver
(batchId=295)
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
(batchId=291)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=284)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriter
(batchId=358)
org.apache.hive.streaming.TestStreaming.testAllTypesDelimitedWriterInputStream
(batchId=358)
org.apache.hive.streaming.TestStreaming.testAutoRollTransactionBatch
(batchId=358)
org.apache.hive.streaming.TestStreaming.testBucketingWhereBucketColIsNotFirstCol
(batchId=358)
org.apache.hive.streaming.TestStreaming.testCommitWithKeyValue (batchId=358)
org.apache.hive.streaming.TestStreaming.testConnectionWithWriteId (batchId=358)
org.apache.hive.streaming.TestStreaming.testInterleavedTransactionBatchCommits
(batchId=358)
org.apache.hive.streaming.TestStreaming.testMultipleTransactionBatchCommits
(batchId=358)
org.apache.hive.streaming.TestStreaming.testNoBuckets (batchId=358)
org.apache.hive.streaming.TestStreaming.testStreamBucketingMatchesRegularBucketing
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPStreamBucketingMatchesRegularBucketing
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevel
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevelMissingPartitionValues
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevelNonStringPartitionColumns
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testJsonInputStreamDP
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testRegexInputStreamDP
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteAfterAbort
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteAfterClose
(batchId=358)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testWriteBeforeBegin
(batchId=358)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18656/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18656/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18656/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980759 - PreCommit-HIVE-Build
> Upgrade HBase dependencies off alpha4 release
> -
>
> Key: HIVE-20205
> URL: https://issues.apache.org/jira/browse/HIVE-20205
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-20205.1.patch, HIVE-20205.1.patch,
> HIVE-20205.2.patch, HIVE-20205.2.patch, HIVE-20205.3.patch, HIVE-20205.patch,
> HIVE-20205.patch
>
>
> Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and
> 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt
> be any API surprises. (but we never know)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[
https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933908#comment-16933908
]
Hive QA commented on HIVE-20205:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18656/dev-support/hive-personality.sh
|
| git revision | master / 5d1a949 |
| Default Java | 1.8.0_111 |
| modules | C: . itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18656/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade HBase dependencies off alpha4 release
> -
>
> Key: HIVE-20205
> URL: https://issues.apache.org/jira/browse/HIVE-20205
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-20205.1.patch, HIVE-20205.1.patch,
> HIVE-20205.2.patch, HIVE-20205.2.patch, HIVE-20205.3.patch, HIVE-20205.patch,
> HIVE-20205.patch
>
>
> Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and
> 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt
> be any API surprises. (but we never know)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Patch Available (was: Open)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Attachment: HIVE-22079.6.patch
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[
https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-22079:
---
Status: Open (was: Patch Available)
> Post order walker for iterating over expression tree
>
>
> Key: HIVE-22079
> URL: https://issues.apache.org/jira/browse/HIVE-22079
> Project: Hive
> Issue Type: Improvement
> Components: Logical Optimizer, Physical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch,
> HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch
>
>
> Current {{DefaultGraphWalker}} is used to iterate over an expression tree.
> This walker uses hash map to keep track of visited/processed nodes. If an
> expression tree is large this adds significant overhead due to map lookup.
> For an expression trees we can instead use post order traversal and avoid
> using map.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22042) Set hive.exec.dynamic.partition.mode=nonstrict by default
[
https://issues.apache.org/jira/browse/HIVE-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933880#comment-16933880
]
Hive QA commented on HIVE-22042:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980648/HIVE-22042.1.patch
{color:green}SUCCESS:{color} +1 due to 291 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 16833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer13]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part11]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part12]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part6]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_2]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez-tag]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=361)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18655/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18655/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18655/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980648 - PreCommit-HIVE-Build
> Set hive.exec.dynamic.partition.mode=nonstrict by default
> -
>
> Key: HIVE-22042
> URL: https://issues.apache.org/jira/browse/HIVE-22042
> Project: Hive
> Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: John Sherman
>Priority: Minor
> Attachments: HIVE-22042.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22042) Set hive.exec.dynamic.partition.mode=nonstrict by default
[
https://issues.apache.org/jira/browse/HIVE-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933873#comment-16933873
]
Hive QA commented on HIVE-22042:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} branch/hcatalog no findbugs output file
(hcatalog/target/findbugsXml.xml) {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} The patch hcatalog passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 536
unchanged - 21 fixed = 536 total (was 557) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
43s{color} | {color:red} patch/hcatalog no findbugs output file
(hcatalog/target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5
[jira] [Commented] (HIVE-17131) Add InterfaceAudience and InterfaceStability annotations for SerDe APIs
[ https://issues.apache.org/jira/browse/HIVE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933856#comment-16933856 ] Ashutosh Chauhan commented on HIVE-17131: - yes for HIVE-16374 for master. > Add InterfaceAudience and InterfaceStability annotations for SerDe APIs > --- > > Key: HIVE-17131 > URL: https://issues.apache.org/jira/browse/HIVE-17131 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-17131.1.branch-2.patch, HIVE-17131.1.patch > > > Adding InterfaceAudience and InterfaceStability annotations for the core > SerDe APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933847#comment-16933847
]
Hive QA commented on HIVE-22201:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980739/HIVE-22201.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980739 - PreCommit-HIVE-Build
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933820#comment-16933820
]
Hive QA commented on HIVE-22201:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18654/dev-support/hive-personality.sh
|
| git revision | master / 5d1a949 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18654/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933760#comment-16933760
]
Hive QA commented on HIVE-22218:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980728/HIVE-22218.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18653/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980728 - PreCommit-HIVE-Build
> Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
>
>
> Key: HIVE-22218
> URL: https://issues.apache.org/jira/browse/HIVE-22218
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22218.01.patch
>
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #5: extract all the workload management related analyzers from
> DDLSemanticAnalyzer, and move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20113: -- Assignee: Vineet Garg (was: Gopal V) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch, > HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch, > HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Patch Available (was: Open) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch, > HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch, > HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Open (was: Patch Available) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch, > HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch, > HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Attachment: HIVE-20113.10.patch > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch, > HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch, > HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933737#comment-16933737
]
Hive QA commented on HIVE-22218:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 146 unchanged - 59
fixed = 147 total (was 205) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} ql generated 0 new + 1570 unchanged - 1 fixed = 1570
total (was 1571) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18653/dev-support/hive-personality.sh
|
| git revision | master / 5d1a949 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18653/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18653/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18653/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
>
>
> Key: HIVE-22218
> URL: https://issues.apache.org/jira/browse/HIVE-22218
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22218.01.patch
>
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #5: extract all the workload management related analyzers from
> DDLSemanticAnalyzer, and move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933697#comment-16933697
]
Dawood commented on HIVE-21237:
---
Thanks [~yumwang] verified with Hive 2.3.6 Hadoop 3.2.0 jdk-11.0.4, Issue is
not reproduced.
{noformat}
[root@hive bin]# $HIVE_HOME/bin/hive
which: no hbase in
(/opt/apache-hive-2.3.6-bin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/java/jdk-11.0.4/bin:/usr/java/jdk-11.0.4/bin:/opt/hadoop-3.2.0/sbin:/opt/hadoop-3.2.0/bin:/usr/java/jdk-11.0.4/bin:/opt/hadoop-3.2.0/sbin:/opt/hadoop-3.2.0/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-2.3.6-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]Logging initialized using
configuration in
jar:file:/opt/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar!/hive-log4j2.properties
Async: true
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.apache.hadoop.hive.common.StringInternUtils
(file:/opt/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar) to field
java.net.URI.string
WARNING: Please consider reporting this to the maintainers of
org.apache.hadoop.hive.common.StringInternUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
Hive-on-MR is deprecated in Hive 2 and may not be available in the future
versions. Consider using a different execution engine (i.e. spark, tez) or
using Hive 1.X releases.
hive>
{noformat}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933675#comment-16933675
]
Hive QA commented on HIVE-22084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980725/HIVE-22084.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16989 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18652/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18652/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18652/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980725 - PreCommit-HIVE-Build
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch,
> HIVE-22084.03.patch, HIVE-22084.04.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-14499) Add HMS metrics for materialized views
[ https://issues.apache.org/jira/browse/HIVE-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-14499: -- Assignee: John Sherman > Add HMS metrics for materialized views > -- > > Key: HIVE-14499 > URL: https://issues.apache.org/jira/browse/HIVE-14499 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: John Sherman >Priority: Major > > As in HIVE-10761/HIVE-12499. > We should be able to show some metrics related to materialized views, such as > the number of materialized views, size of the materialized views, number of > accesses, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21267) Extend HiveRelColumnsAlignment to reorder group-by and join keys on decreasing NDV automatically
[
https://issues.apache.org/jira/browse/HIVE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-21267:
--
Assignee: John Sherman
> Extend HiveRelColumnsAlignment to reorder group-by and join keys on
> decreasing NDV automatically
>
>
> Key: HIVE-21267
> URL: https://issues.apache.org/jira/browse/HIVE-21267
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: John Sherman
>Priority: Major
> Labels: newbie++
>
> {{HiveRelColumnsAlignment}} was introduced to align the order of columns in
> join, group-by, and order-by operators in the plan pipeline, trying to
> increase the effect of ReduceDeduplication and thus reducing data shuffle.
> The optimization could be extended to reorder group-by and join keys on
> decreasing NDV, which would accelerate comparison runtime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22213) TxnHander cleanupRecords should only clean records belonging to default catalog
[ https://issues.apache.org/jira/browse/HIVE-22213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22213: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master > TxnHander cleanupRecords should only clean records belonging to default > catalog > --- > > Key: HIVE-22213 > URL: https://issues.apache.org/jira/browse/HIVE-22213 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22213.1.patch, HIVE-22213.2.patch > > > Currently it removes record for given database and given table without > checking for the catalog, as a result it can end up removing records when it > shouldn't. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Status: Patch Available (was: Open) > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Status: Open (was: Patch Available) > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Attachment: HIVE-22205.3.patch > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22084:
-
Attachment: HIVE-22084.04.patch
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch,
> HIVE-22084.03.patch, HIVE-22084.04.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933630#comment-16933630
]
Hive QA commented on HIVE-22084:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 17 unchanged - 0 fixed
= 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 1 new + 1571 unchanged - 0 fixed = 1572
total (was 1571) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of partitions, which is known to be non-null in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.exchange_partition(Map,
String, String, String, String, String, String) Redundant null check at
SessionHiveMetaStoreClient.java:is known to be non-null in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.exchange_partition(Map,
String, String, String, String, String, String) Redundant null check at
SessionHiveMetaStoreClient.java:[line 1590] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18652/dev-support/hive-personality.sh
|
| git revision | master / 5b4deef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18652/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18652/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18652/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933627#comment-16933627
]
Dawood commented on HIVE-21237:
---
[~yumwang] The rest of the version can be same with Hive 2.3.6? Hadoop 3.2.0,
jdk-11.0.4?
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-21975:
Attachment: HIVE-21975.patch
> Fix incremental compilation
> ---
>
> Key: HIVE-21975
> URL: https://issues.apache.org/jira/browse/HIVE-21975
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Steve Carlin
>Priority: Major
> Attachments: HIVE-21975.patch
>
>
> we have an incremental compilation issue around SA ; mostly because of "?
> extends Serializable"
> it could be reproduced with:
> {code}
> git clean -dfx
> mvn install -pl ql -am -DskipTests
> touch `find . -name Sema*A*java` `find . -name Task*Factory.java`
> mvn install -pl ql -DskipTests
> {code}
> error is:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60]
> incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to
> java.util.List>
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49]
> incompatible types: java.util.List>
> cannot be converted to java.util.List extends java.io.Serializable>>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21975) Fix incremental compilation
[
https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-21975:
Status: Patch Available (was: Open)
> Fix incremental compilation
> ---
>
> Key: HIVE-21975
> URL: https://issues.apache.org/jira/browse/HIVE-21975
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Steve Carlin
>Priority: Major
> Attachments: HIVE-21975.patch
>
>
> we have an incremental compilation issue around SA ; mostly because of "?
> extends Serializable"
> it could be reproduced with:
> {code}
> git clean -dfx
> mvn install -pl ql -am -DskipTests
> touch `find . -name Sema*A*java` `find . -name Task*Factory.java`
> mvn install -pl ql -DskipTests
> {code}
> error is:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hive-exec: Compilation failure: Compilation
> failure:
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60]
> incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to
> java.util.List>
> [ERROR]
> /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49]
> incompatible types: java.util.List>
> cannot be converted to java.util.List extends java.io.Serializable>>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933602#comment-16933602
]
Yuming Wang commented on HIVE-21237:
[~dawood.m] It seems related to HIVE-21584. Could you try Hive 2.3.6?
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-22209:
Attachment: HIVE-22209.patch
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
> Attachments: HIVE-22209.patch
>
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-20205: - Status: Patch Available (was: Open) > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.1.patch, HIVE-20205.1.patch, > HIVE-20205.2.patch, HIVE-20205.2.patch, HIVE-20205.3.patch, HIVE-20205.patch, > HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-20205: - Attachment: HIVE-20205.3.patch > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.1.patch, HIVE-20205.1.patch, > HIVE-20205.2.patch, HIVE-20205.2.patch, HIVE-20205.3.patch, HIVE-20205.patch, > HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20205) Upgrade HBase dependencies off alpha4 release
[ https://issues.apache.org/jira/browse/HIVE-20205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-20205: - Status: Open (was: Patch Available) > Upgrade HBase dependencies off alpha4 release > - > > Key: HIVE-20205 > URL: https://issues.apache.org/jira/browse/HIVE-20205 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-20205.1.patch, HIVE-20205.1.patch, > HIVE-20205.2.patch, HIVE-20205.2.patch, HIVE-20205.patch, HIVE-20205.patch > > > Appears Hive has dependencies on hbase 2.0.0-alpha4 releases. HBase 2.0.0 and > 2.0.1 have been released. HBase team recommends 2.0.1 and says there shouldnt > be any API surprises. (but we never know) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22214) Explain vectorization should disable user level explain
[ https://issues.apache.org/jira/browse/HIVE-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933565#comment-16933565 ] Gopal V commented on HIVE-22214: LGTM - +1 tests pending > Explain vectorization should disable user level explain > --- > > Key: HIVE-22214 > URL: https://issues.apache.org/jira/browse/HIVE-22214 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-22214.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[
https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Carlin updated HIVE-22209:
Status: Patch Available (was: Open)
> Creating a materialized view with no tables should be handled more gracefully
> -
>
> Key: HIVE-22209
> URL: https://issues.apache.org/jira/browse/HIVE-22209
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Steve Carlin
>Priority: Minor
>
> Currently, materialized views without a table reference are not supported.
> However, instead of printing a clear message about it, when a materialized
> view is created without a table reference, we fail with an unclear message.
> {code}
> > create materialized view mv_test1 as select 5;
> (...)
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request
> failed :
> INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) )
> INFO : Completed executing
> command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a);
> Time taken: 10.469 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep
> tion(message:Add request failed : INSERT INTO MV_TABLES_USED
> (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22215) Compaction of sorted table
[
https://issues.apache.org/jira/browse/HIVE-22215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933545#comment-16933545
]
Rajkumar Singh commented on HIVE-22215:
---
[~pasza] this is as per design, hive skip the compaction for sorted bucket table
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java#L151]
> Compaction of sorted table
> --
>
> Key: HIVE-22215
> URL: https://issues.apache.org/jira/browse/HIVE-22215
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Pawel Jurkiewicz
>Priority: Major
>
> I recently came across an issue regarding compacting tables with sorting.
> I am creating and populating with test data two tables: both ACID but only
> one is sorted
> {code:sql}
> USE priv;
> DROP TABLE IF EXISTS test_data;
> DROP TABLE IF EXISTS test_compact_insert_with_sorting;
> DROP TABLE IF EXISTS test_compact_insert_without_sorting;
> CREATE TABLE test_data AS SELECT 'foobar' col;
> CREATE TABLE test_compact_insert_with_sorting (col string)
> CLUSTERED BY (col) SORTED BY (col) INTO 1 BUCKETS
> TBLPROPERTIES ('transactional' = 'true',
> 'transactional_properties'='insert_only');
> CREATE TABLE test_compact_insert_without_sorting (col string)
> CLUSTERED BY (col) INTO 1 BUCKETS
> TBLPROPERTIES ('transactional' = 'true',
> 'transactional_properties'='insert_only');
> INSERT OVERWRITE TABLE test_compact_insert_with_sorting SELECT col FROM
> test_data;
> INSERT OVERWRITE TABLE test_compact_insert_without_sorting SELECT col FROM
> test_data; INSERT OVERWRITE TABLE test_compact_insert_with_sorting SELECT
> col FROM test_data;
> INSERT OVERWRITE TABLE test_compact_insert_without_sorting SELECT col FROM
> test_data;
> {code}
> As expected, after these operations two base files were created for each
> table:
> {code:bash}
> $ hdfs dfs -ls /warehouse/tablespace/managed/hive/priv.db/test_compact_insert*
> Found 2 items
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_with_sorting/base_001
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_with_sorting/base_002
> Found 2 items
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_without_sorting/base_001
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_without_sorting/base_002
> {code}
> But after running manual compaction on those tables:
> {code:sql}
> USE priv;
> ALTER TABLE test_compact_insert_with_sorting COMPACT 'MAJOR';
> ALTER TABLE test_compact_insert_without_sorting COMPACT 'MAJOR';
> {code}
> Tuns out only the one without sorting got compacted:
> {code:bash}
> hdfs dfs -ls /warehouse/tablespace/managed/hive/priv.db/test_compact*
> Found 2 items
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_with_sorting/base_001
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_with_sorting/base_002
> Found 1 items
> drwxrwx---+ - hive hadoop 0 2019-09-18 15:08
> /warehouse/tablespace/managed/hive/priv.db/test_compact_insert_without_sorting/base_002
> {code}
> Compactions inspection returns:
> {code:bash}
> $ beeline -e 'show compactions' | grep priv | grep test_compact
> | 7598474 | priv | test_compact_insert_with_sorting | ---
> | MAJOR | succeeded |
> master-01.pd.my-domain.com.pl-51 | 1568812155386 | 11 | None
> |
> | 7598475 | priv | test_compact_insert_without_sorting| ---
> | MAJOR | succeeded | ---
>| 1568812155403 | 298| None
> {code}
> Is this by design? Both compactions states are 'succeeded' but only the one
> that resulted in reducing number of base files took some time. Another
> remarkable behavior is compaction of the table with sorting has worker
> assigned meaning it is still in progress?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933524#comment-16933524
]
Hive QA commented on HIVE-22197:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980722/HIVE-22197.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16834 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18651/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18651/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18651/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980722 - PreCommit-HIVE-Build
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch,
> HIVE-22197.03.patch, HIVE-22197.04.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[
https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933500#comment-16933500
]
Dawood commented on HIVE-21237:
---
Hi [~yumwang]
I am seeing the same issue on my setup
Hadoop 3.2.0, apache-hive-3.1.2, jdk-11.0.4
how to resolve this?
{noformat}
[root@hadoop ~]# hive version
which: no hbase in
(/opt/apache-hive-3.1.2-bin/bin:/opt/apache-hive-3.1.2-bin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/jdk-11.0.4/bin:/root/bin:/usr/java/jdk-11.0.4/bin:/opt/hadoop-3.2.0/sbin:/opt/hadoop-3.2.0/bin:/usr/java/jdk-11.0.4/bin:/opt/hadoop-3.2.0/sbin:/opt/hadoop-3.2.0/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive Session ID = c5f9e81f-6c8b-4c83-a924-9a57497087c5
Exception in thread "main" java.lang.ClassCastException: class
jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
java.net.URLClassLoader are in module java.base of loader 'bootstrap')
at
org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:413)
at
org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:389)
at
org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
{noformat}
> [JDK 11] SessionState can't be initialized due to classloader problem
> -
>
> Key: HIVE-21237
> URL: https://issues.apache.org/jira/browse/HIVE-21237
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.1.1
> Environment: JDK11, Hadoop-3, Hive 3.1.1
>Reporter: Uma Maheswara Rao G
>Priority: Major
>
> When I start Hive with JDK11
> {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID =
> cecd9c34-d61a-44d0-9e52-a0a7d6413e49
> Exception in thread "main" java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410)
> at
> org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386)
> at
> org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22042) Set hive.exec.dynamic.partition.mode=nonstrict by default
[ https://issues.apache.org/jira/browse/HIVE-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22042: Status: Patch Available (was: In Progress) > Set hive.exec.dynamic.partition.mode=nonstrict by default > - > > Key: HIVE-22042 > URL: https://issues.apache.org/jira/browse/HIVE-22042 > Project: Hive > Issue Type: Bug >Reporter: Jesus Camacho Rodriguez >Assignee: John Sherman >Priority: Minor > Attachments: HIVE-22042.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933464#comment-16933464
]
Hive QA commented on HIVE-22197:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 29 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18651/dev-support/hive-personality.sh
|
| git revision | master / 5b4deef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18651/yetus/whitespace-tabs.txt
|
| modules | C: shims/0.23 ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18651/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch,
> HIVE-22197.03.patch, HIVE-22197.04.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933406#comment-16933406
]
Hive QA commented on HIVE-21508:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980710/HIVE-21508.6.branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 135 failed/errored test(s), 14435 tests
executed
*Failed tests:*
{noformat}
TestAddPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestAddPartitionsFromPartSpec - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestAggregateStatsCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestAlterPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestAppendPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=275)
TestCachedStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogCaching - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestCatalogNonDefaultClient - did not produce a TEST-*.xml file (likely timed
out) (batchId=228)
TestCatalogNonDefaultSvr - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestCatalogOldClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCatalogs - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestCheckConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestCloseableThreadLocal - did not produce a TEST-*.xml file (likely timed out)
(batchId=334)
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestDatabases - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDeadline - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestDefaultConstraint - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestDropPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=275)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=231)
TestExchangePartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed
out) (batchId=239)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestForeignKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestFunctions - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestGetPartitions - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestGetPartitionsUsingProjectionAndFilterSpecs - did not produce a TEST-*.xml
file (likely timed out) (batchId=230)
TestGetTableMeta - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHdfsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
TestHiveAlterHandler - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHiveMetaStoreGetMetaConf - did not produce a TEST-*.xml file (likely timed
out) (batchId=238)
TestHiveMetaStorePartitionSpecs - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
TestHiveMetaStoreSchemaMethods - did not produce a TEST-*.xml file (likely
timed out) (batchId=238)
TestHiveMetaStoreTimeout - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=238)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=233)
TestHiveMetaToolCommandLine - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestHiveMetastoreCli - did not produce a TEST-*.xml file (likely timed out)
(batchId=228)
TestHmsServerAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out)
(batchId=239)
TestJSONMessageDeserializer - did not produce a TEST-*.xml file (likely timed
out) (batchId=234)
TestKuduCliDriver - did not
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[
https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933398#comment-16933398
]
Naveen Gangam commented on HIVE-22205:
--
Hi [~tamaas] So after our discussion yesterday, I found a class
{{org.apache.hive.testutils.MiniZooKeeperCluster}} in the hive code base, that
was not being used much. Looked similar to the one we were using from Hbase
dependencies. I replaced all usages of this class with the one I found in hive
code base and tests started working (there were some failures but not related
to the ZK server startup). So atleast tests were running. I had figured that
the failures were related to somehow older hbase code.
It makes sense given what you said. I will submit a patch with upgraded HBase
dependency and see how the tests do. Thanks
> Upgrade zookeeper and curator versions
> --
>
> Key: HIVE-22205
> URL: https://issues.apache.org/jira/browse/HIVE-22205
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
> Attachments: HIVE-22205.2.patch, HIVE-22205.patch
>
>
> Other components like hadoop have switched to using new ZK versions. So these
> jars end up in classpath for hive services and could cause issues due to
> in-compatible curator versions that hive uses.
> So it makes sense for hive to upgrade the ZK and curator versions to try to
> keep up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Mishra updated HIVE-22201:
---
Status: Open (was: Patch Available)
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Mishra updated HIVE-22201:
---
Attachment: HIVE-22201.05.patch
Status: Patch Available (was: Open)
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch, HIVE-22201.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22203) Break up DDLSemanticAnalyzer - extract Process related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22203?focusedWorklogId=315052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315052 ] ASF GitHub Bot logged work on HIVE-22203: - Author: ASF GitHub Bot Created on: 19/Sep/19 13:39 Start Date: 19/Sep/19 13:39 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #774: HIVE-22203 Break up DDLSemanticAnalyzer - extract Process related analyzers URL: https://github.com/apache/hive/pull/774 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315052) Time Spent: 20m (was: 10m) > Break up DDLSemanticAnalyzer - extract Process related analyzers > > > Key: HIVE-22203 > URL: https://issues.apache.org/jira/browse/HIVE-22203 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22203.01.patch, HIVE-22203.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #4: extract all the process related analyzers from DDLSemanticAnalyzer, > and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22218: -- Attachment: HIVE-22218.01.patch > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22218.01.patch > > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #5: extract all the workload management related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22218: -- Status: Patch Available (was: Open) > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22218.01.patch > > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #5: extract all the workload management related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22218: -- Description: DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to refactor it in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: * have a separate class for each analyzers * have a package for each operation, containing an analyzer, a description, and an operation, so the amount of classes under a package is more manageable Step #5: extract all the workload management related analyzers from DDLSemanticAnalyzer, and move them under the new package. was: DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to refactor it in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: * have a separate class for each analyzers * have a package for each operation, containing an analyzer, a description, and an operation, so the amount of classes under a package is more manageable Step #4: extract all the process related analyzers from DDLSemanticAnalyzer, and move them under the new package. > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #5: extract all the workload management related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22203) Break up DDLSemanticAnalyzer - extract Process related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22203: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Break up DDLSemanticAnalyzer - extract Process related analyzers > > > Key: HIVE-22203 > URL: https://issues.apache.org/jira/browse/HIVE-22203 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22203.01.patch, HIVE-22203.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #4: extract all the process related analyzers from DDLSemanticAnalyzer, > and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933342#comment-16933342
]
Hive QA commented on HIVE-21508:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-18650/patches/PreCommit-HIVE-Build-18650.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18650/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.6.branch-3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-22218: - > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #4: extract all the process related analyzers from DDLSemanticAnalyzer, > and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22084:
-
Attachment: HIVE-22084.03.patch
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch,
> HIVE-22084.03.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1697#comment-1697
]
Hive QA commented on HIVE-22084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980699/HIVE-22084.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16989 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=362)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18649/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18649/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18649/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980699 - PreCommit-HIVE-Build
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Patch Available (was: Open) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: HIVE-22197.04.patch > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: (was: HIVE-22197.04.patch) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Open (was: Patch Available) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933306#comment-16933306
]
Hive QA commented on HIVE-22084:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
178 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 1 new + 17 unchanged - 0 fixed
= 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 1 new + 1571 unchanged - 0 fixed = 1572
total (was 1571) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of partitions, which is known to be non-null in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.exchange_partition(Map,
String, String, String, String, String, String) Redundant null check at
SessionHiveMetaStoreClient.java:is known to be non-null in
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.exchange_partition(Map,
String, String, String, String, String, String) Redundant null check at
SessionHiveMetaStoreClient.java:[line 1590] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18649/dev-support/hive-personality.sh
|
| git revision | master / f7d5ef2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18649/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18649/yetus/new-findbugs-ql.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18649/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933282#comment-16933282
]
Hive QA commented on HIVE-22197:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980680/HIVE-22197.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 16834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createGetDrop[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPkInOtherCatalog[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.createTableWithConstraintsPk[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.doubleAddUniqueConstraint[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchCatalog[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchDb[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.getNoSuchTable[Remote]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.inOtherCatalog[Remote]
(batchId=225)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18648/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18648/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18648/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980680 - PreCommit-HIVE-Build
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch,
> HIVE-22197.03.patch, HIVE-22197.04.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933278#comment-16933278 ] Tamas Penzes commented on HIVE-22205: - [~ngangam] Hive upstream master uses a very old HBase version (2.0.0-alpha4) see: [https://github.com/apache/hive/blob/master/pom.xml#L158] . In HBase code we set up a ZooKeeper minicluster the following way: [https://github.com/apache/hbase/blob/rel/2.0.0/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java#L172] But from ZooKeeper 3.5.3-beta we have to whitelist the four letter words we want to use. See [https://zookeeper.apache.org/doc/r3.5.5/zookeeperAdmin.html#sc_clusterOptions] In HBase we have patched this test setup in HBASE-21203, which means you have to use HBase version [2.2.0|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.2.0], [2.1.1|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.1.1], [2.0.3|https://issues.apache.org/jira/issues/?jql=project+%3D+HBASE+AND+fixVersion+%3D+2.0.3] or above. There we have the following line added to test setup: [https://github.com/apache/hbase/blob/master/hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java#L428] This way HBase testing and Hive should work with the new ZooKeeper (and Curator) together. Even without this ticket I would any way support the idea using a newer version of HBase and not an alpha version. HBase 2.0.x is deprecated anyway, so please use 2.1.x or 2.2.1 from Hive. > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933259#comment-16933259
]
Ana Jalba commented on HIVE-21508:
--
Thanks, [~pvary]! And no worries, you've been really helpful with this :)
I've just added the last patch.
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.6.branch-3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ana Jalba updated HIVE-21508:
-
Attachment: HIVE-21508.6.branch-3.patch
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.6.branch-3.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[
https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933253#comment-16933253
]
Hive QA commented on HIVE-22197:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 29 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18648/dev-support/hive-personality.sh
|
| git revision | master / f7d5ef2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18648/yetus/whitespace-tabs.txt
|
| modules | C: shims/0.23 ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18648/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Common Merge join throwing class cast exception
>
>
> Key: HIVE-22197
> URL: https://issues.apache.org/jira/browse/HIVE-22197
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch,
> HIVE-22197.03.patch, HIVE-22197.04.patch
>
>
> In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed
> and stored in the writable format, but the object inspector is initialized to
> default. So when join operator is fetching the data from dummy store
> operator, its getting the OI is Long and the row as LongWritable. This is
> causing the class cast exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933247#comment-16933247
]
Peter Vary commented on HIVE-21508:
---
Sorry [~ananamj]! Lost the track of this jira :(
Committed to 3.1. Can not set the fix version to 3.1.3, since that is not
available yet.
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933224#comment-16933224
]
Hive QA commented on HIVE-22201:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980673/HIVE-22201.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[updateBasicStats]
(batchId=69)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18647/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18647/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18647/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980673 - PreCommit-HIVE-Build
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-17131) Add InterfaceAudience and InterfaceStability annotations for SerDe APIs
[ https://issues.apache.org/jira/browse/HIVE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933217#comment-16933217 ] Peter Vary commented on HIVE-17131: --- [~ashutoshc]: Do we plan to move forward with HIVE-16374, or it might worth to push this to master as well? Thanks, Peter > Add InterfaceAudience and InterfaceStability annotations for SerDe APIs > --- > > Key: HIVE-17131 > URL: https://issues.apache.org/jira/browse/HIVE-17131 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 2.4.0 > > Attachments: HIVE-17131.1.branch-2.patch, HIVE-17131.1.patch > > > Adding InterfaceAudience and InterfaceStability annotations for the core > SerDe APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[
https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933210#comment-16933210
]
Ana Jalba commented on HIVE-21508:
--
[~pvary], is there any way I can help with the patch for branch-3.1 or should I
continue uploading the last one (for branch-3)?
> ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
> --
>
> Key: HIVE-21508
> URL: https://issues.apache.org/jira/browse/HIVE-21508
> Project: Hive
> Issue Type: Bug
> Components: Clients
>Affects Versions: 3.2.0, 2.3.4
>Reporter: Adar Dembo
>Assignee: Ana Jalba
>Priority: Major
> Fix For: 2.4.0, 4.0.0, 2.3.7
>
> Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch,
> HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch,
> HIVE-21508.5.branch-3.1.patch, HIVE-21508.patch
>
>
> There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called
> from the constructor) on master:
> {noformat}
> private URI metastoreUris[];
> ...
> if (MetastoreConf.getVar(conf,
> ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) {
> List uriList = Arrays.asList(metastoreUris);
> Collections.shuffle(uriList);
> metastoreUris = (URI[]) uriList.toArray();
> }
> {noformat}
> The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10,
> possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to
> {{RANDOM}} so this should affect anyone who creates a
> {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}}
> to avoid the broken case; I'm working against 2.3.4 where there's no such
> workaround.
> [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9]
> a StackOverflow post that explains the issue in more detail. Interestingly,
> the author described the issue in the context of the HMS; not sure why there
> was no follow up with a Hive bug report.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22084) Implement exchange partitions related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22084:
-
Attachment: HIVE-22084.02.patch
> Implement exchange partitions related methods on temporary tables
> -
>
> Key: HIVE-22084
> URL: https://issues.apache.org/jira/browse/HIVE-22084
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22084.01.patch, HIVE-22084.02.patch
>
>
> IMetaStoreClient exposes the following methods related to exchanging
> partitions:
> {code:java}
> Partition exchange_partition(Map partitionSpecs, String
> sourceDb, String sourceTable, String destdb, String destTableName);
> Partition exchange_partition(Map partitionSpecs, String
> sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceDb, String sourceTable, String destdb, String destTableName);
> List exchange_partitions(Map partitionSpecs,
> String sourceCat, String sourceDb, String sourceTable, String destCat, String
> destdb, String destTableName);{code}
> In order to support partitions on temporary tables, these methods must be
> implemented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933195#comment-16933195 ] Ivan Suller commented on HIVE-21000: Accumulo 2.0.0 has been finally released: https://accumulo.apache.org/release/accumulo-2.0.0/ Unfortunately this patch is no longer appliable, so I have to redone it from scratch. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22201) ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws ArrayIndexOutOfBoundsException if no big table is selected
[
https://issues.apache.org/jira/browse/HIVE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933189#comment-16933189
]
Hive QA commented on HIVE-22201:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18647/dev-support/hive-personality.sh
|
| git revision | master / f7d5ef2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18647/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ConvertJoinMapJoin#checkShuffleSizeForLargeTable throws
> ArrayIndexOutOfBoundsException if no big table is selected
> --
>
> Key: HIVE-22201
> URL: https://issues.apache.org/jira/browse/HIVE-22201
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Himanshu Mishra
>Assignee: Himanshu Mishra
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22201.01.patch, HIVE-22201.02.patch,
> HIVE-22201.03.patch, HIVE-22201.04.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In case when {{bigTableCandidateSet}} is empty e.g. full outer join, we end
> up calling {{checkShuffleSizeForLargeTable}} with {{bigTablePosition}} as -1,
> resulting in {{ArrayIndexOutOfBoundsException}}.
> Also, should we return as soon as we see {{bigTableCandidateSet}} is empty ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22163) CBO: Enabling CBO turns on stats estimation, even when the estimation is disabled
[
https://issues.apache.org/jira/browse/HIVE-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933164#comment-16933164
]
Hive QA commented on HIVE-22163:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980668/HIVE-22163.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16834 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18646/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18646/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18646/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980668 - PreCommit-HIVE-Build
> CBO: Enabling CBO turns on stats estimation, even when the estimation is
> disabled
> -
>
> Key: HIVE-22163
> URL: https://issues.apache.org/jira/browse/HIVE-22163
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22163.1.patch, HIVE-22163.1.patch,
> HIVE-22163.1.patch, HIVE-22163.2.patch, HIVE-22163.3.patch,
> HIVE-22163.4.patch, HIVE-22163.4.patch, HIVE-22163.5.patch,
> HIVE-22163.5.patch, HIVE-22163.5.patch, HIVE-22163.5.patch,
> HIVE-22163.5.patch, HIVE-22163.5.patch, HIVE-22163.5.patch
>
>
> {code}
> create table claims(claim_rec_id bigint, claim_invoice_num string, typ_c int);
> alter table claims update statistics set
> ('numRows'='1154941534','rawDataSize'='1135307527922');
> set hive.stats.estimate=false;
> explain extended select count(1) from claims where typ_c=3;
> set hive.stats.ndv.estimate.percent=5e-7;
> explain extended select count(1) from claims where typ_c=3;
> {code}
> Expecting the standard /2 for the single filter, but we instead get 5 rows.
> {code}
> 'Map Operator Tree:'
> 'TableScan'
> ' alias: claims'
> ' filterExpr: (typ_c = 3) (type: boolean)'
> ' Statistics: Num rows: 1154941534 Data size: 4388777832
> Basic stats: COMPLETE Column stats: NONE'
> ' GatherStats: false'
> ' Filter Operator'
> 'isSamplingPred: false'
> 'predicate: (typ_c = 3) (type: boolean)'
> 'Statistics: Num rows: 5 Data size: 19 Basic stats:
> COMPLETE Column stats: NONE'
> {code}
> The estimation is in effect, as changing the estimate.percent changes this.
> {code}
> ' filterExpr: (typ_c = 3) (type: boolean)'
> ' Statistics: Num rows: 1154941534 Data size: 4388777832
> Basic stats: COMPLETE Column stats: NONE'
> ' GatherStats: false'
> ' Filter Operator'
> 'isSamplingPred: false'
> 'predicate: (typ_c = 3) (type: boolean)'
> 'Statistics: Num rows: 230988307 Data size: 877755567
> Basic stats: COMPLETE Column stats: NONE'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22195) Configure authentication type for Zookeeper when different from the default cluster wide
[ https://issues.apache.org/jira/browse/HIVE-22195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933145#comment-16933145 ] Peter Vary commented on HIVE-22195: --- +1 > Configure authentication type for Zookeeper when different from the default > cluster wide > > > Key: HIVE-22195 > URL: https://issues.apache.org/jira/browse/HIVE-22195 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22195.1.patch, HIVE-22195.2.patch, > HIVE-22195.4.patch, HIVE-22195.5.patch, HIVE-22195.6.patch, > HIVE-22195.7.patch, HIVE-22195.8.patch > > > This could be useful in case when cluster is kerberized, but Zookeeper is not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22163) CBO: Enabling CBO turns on stats estimation, even when the estimation is disabled
[
https://issues.apache.org/jira/browse/HIVE-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933135#comment-16933135
]
Hive QA commented on HIVE-22163:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18646/dev-support/hive-personality.sh
|
| git revision | master / f7d5ef2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18646/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> CBO: Enabling CBO turns on stats estimation, even when the estimation is
> disabled
> -
>
> Key: HIVE-22163
> URL: https://issues.apache.org/jira/browse/HIVE-22163
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Reporter: Gopal V
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22163.1.patch, HIVE-22163.1.patch,
> HIVE-22163.1.patch, HIVE-22163.2.patch, HIVE-22163.3.patch,
> HIVE-22163.4.patch, HIVE-22163.4.patch, HIVE-22163.5.patch,
> HIVE-22163.5.patch, HIVE-22163.5.patch, HIVE-22163.5.patch,
> HIVE-22163.5.patch, HIVE-22163.5.patch, HIVE-22163.5.patch
>
>
> {code}
> create table claims(claim_rec_id bigint, claim_invoice_num string, typ_c int);
> alter table claims update statistics set
> ('numRows'='1154941534','rawDataSize'='1135307527922');
> set hive.stats.estimate=false;
> explain extended select count(1) from claims where typ_c=3;
> set hive.stats.ndv.estimate.percent=5e-7;
> explain extended select count(1) from claims where typ_c=3;
> {code}
> Expecting the standard /2 for the single filter, but we instead get 5 rows.
> {code}
> 'Map Operator Tree:'
> 'TableScan'
> ' alias: claims'
> ' filterExpr: (typ_c = 3) (type: boolean)'
> ' Statistics: Num rows: 1154941534 Data size: 4388777832
> Basic stats: COMPLETE Column stats: NONE'
> ' GatherStats: false'
> ' Filter Operator'
> 'isSamplingPred: false'
> 'predicate: (typ_c = 3)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Patch Available (was: Open) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: HIVE-22197.04.patch > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch, HIVE-22197.04.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Open (was: Patch Available) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: (was: HIVE-22197.04.patch) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.02.patch, > HIVE-22197.03.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
