[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118339#comment-17118339
]
Hive QA commented on HIVE-23468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004184/HIVE-23468.8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=130)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22660/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22660/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22660/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004184 - PreCommit-HIVE-Build
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch,
> HIVE-23468.6.patch, HIVE-23468.7.patch, HIVE-23468.8.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23435) Full outer join result is missing rows
[
https://issues.apache.org/jira/browse/HIVE-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman updated HIVE-23435:
Attachment: HIVE-23435.patch
> Full outer join result is missing rows
> ---
>
> Key: HIVE-23435
> URL: https://issues.apache.org/jira/browse/HIVE-23435
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23435.patch
>
>
> Full Outer join result has missing rows. Appears to be a bug with the full
> outer join logic. Expected output is receiving when we do a left and right
> outer join.
> Reproducible steps are mentioned below.
> ~~
> SUPPORT ANALYSIS
> Steps to Reproduce:
> 1. Create a table and insert data:
> create table x (z char(5), x int, y int);
> insert into x values ('one', 1, 50),
> ('two', 2, 30),
> ('three', 3, 30),
> ('four', 4, 60),
> ('five', 5, 70),
> ('six', 6, 80);
> 2. Try full outer with the below command. The result is incomplete, it is
> missing the row:
> NULL NULL NULL three 3 30.0
> Full Outer Join:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 full outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> --+
> x1.z x1.x x1.y x2.z x2.x x2.y
> --+
> one 1 50 NULL NULL NULL
> NULL NULL NULL one 1 50
> two 2 30 NULL NULL NULL
> NULL NULL NULL two 2 30
> three 3 30 NULL NULL NULL
> four 4 60 NULL NULL NULL
> NULL NULL NULL four 4 60
> five 5 70 NULL NULL NULL
> NULL NULL NULL five 5 70
> six 6 80 NULL NULL NULL
> NULL NULL NULL six 6 80
> --+
> 3. Expected output is coming when we use left/right join + union:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 left outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`)
> union
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 right outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> +
> z x y _col3 _col4 _col5
> +
> NULL NULL NULL five 5 70
> NULL NULL NULL four 4 60
> NULL NULL NULL one 1 50
> four 4 60 NULL NULL NULL
> one 1 50 NULL NULL NULL
> six 6 80 NULL NULL NULL
> three 3 30 NULL NULL NULL
> two 2 30 NULL NULL NULL
> NULL NULL NULL six 6 80
> NULL NULL NULL three 3 30
> NULL NULL NULL two 2 30
> five 5 70 NULL NULL NULL
> +
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23435) Full outer join result is missing rows
[
https://issues.apache.org/jira/browse/HIVE-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman updated HIVE-23435:
Status: Patch Available (was: Open)
> Full outer join result is missing rows
> ---
>
> Key: HIVE-23435
> URL: https://issues.apache.org/jira/browse/HIVE-23435
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23435.patch
>
>
> Full Outer join result has missing rows. Appears to be a bug with the full
> outer join logic. Expected output is receiving when we do a left and right
> outer join.
> Reproducible steps are mentioned below.
> ~~
> SUPPORT ANALYSIS
> Steps to Reproduce:
> 1. Create a table and insert data:
> create table x (z char(5), x int, y int);
> insert into x values ('one', 1, 50),
> ('two', 2, 30),
> ('three', 3, 30),
> ('four', 4, 60),
> ('five', 5, 70),
> ('six', 6, 80);
> 2. Try full outer with the below command. The result is incomplete, it is
> missing the row:
> NULL NULL NULL three 3 30.0
> Full Outer Join:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 full outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> --+
> x1.z x1.x x1.y x2.z x2.x x2.y
> --+
> one 1 50 NULL NULL NULL
> NULL NULL NULL one 1 50
> two 2 30 NULL NULL NULL
> NULL NULL NULL two 2 30
> three 3 30 NULL NULL NULL
> four 4 60 NULL NULL NULL
> NULL NULL NULL four 4 60
> five 5 70 NULL NULL NULL
> NULL NULL NULL five 5 70
> six 6 80 NULL NULL NULL
> NULL NULL NULL six 6 80
> --+
> 3. Expected output is coming when we use left/right join + union:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 left outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`)
> union
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 right outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> +
> z x y _col3 _col4 _col5
> +
> NULL NULL NULL five 5 70
> NULL NULL NULL four 4 60
> NULL NULL NULL one 1 50
> four 4 60 NULL NULL NULL
> one 1 50 NULL NULL NULL
> six 6 80 NULL NULL NULL
> three 3 30 NULL NULL NULL
> two 2 30 NULL NULL NULL
> NULL NULL NULL six 6 80
> NULL NULL NULL three 3 30
> NULL NULL NULL two 2 30
> five 5 70 NULL NULL NULL
> +
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Attachment: HIVE-23347.10.patch > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.10.patch, > HIVE-23347.2.patch, HIVE-23347.3.patch, HIVE-23347.4.patch, > HIVE-23347.5.patch, HIVE-23347.6.patch, HIVE-23347.7.patch, > HIVE-23347.8.patch, HIVE-23347.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Patch Available (was: Open) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.10.patch, > HIVE-23347.2.patch, HIVE-23347.3.patch, HIVE-23347.4.patch, > HIVE-23347.5.patch, HIVE-23347.6.patch, HIVE-23347.7.patch, > HIVE-23347.8.patch, HIVE-23347.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Open (was: Patch Available) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch, > HIVE-23347.6.patch, HIVE-23347.7.patch, HIVE-23347.8.patch, HIVE-23347.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Attachment: (was: hive-23361.11.patch) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.10.patch, > hive-23361.11.patch, hive-23361.2.patch, hive-23361.3.patch, > hive-23361.4.patch, hive-23361.5.patch, hive-23361.6.patch, > hive-23361.7.patch, hive-23361.8.patch, hive-23361.9.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118326#comment-17118326
]
Hive QA commented on HIVE-23468:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22660/dev-support/hive-personality.sh
|
| git revision | master / 0b09d01 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22660/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch,
> HIVE-23468.6.patch, HIVE-23468.7.patch, HIVE-23468.8.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by
[jira] [Assigned] (HIVE-23435) Full outer join result is missing rows
[
https://issues.apache.org/jira/browse/HIVE-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman reassigned HIVE-23435:
---
Assignee: Mustafa Iman
> Full outer join result is missing rows
> ---
>
> Key: HIVE-23435
> URL: https://issues.apache.org/jira/browse/HIVE-23435
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Mustafa Iman
>Priority: Major
>
> Full Outer join result has missing rows. Appears to be a bug with the full
> outer join logic. Expected output is receiving when we do a left and right
> outer join.
> Reproducible steps are mentioned below.
> ~~
> SUPPORT ANALYSIS
> Steps to Reproduce:
> 1. Create a table and insert data:
> create table x (z char(5), x int, y int);
> insert into x values ('one', 1, 50),
> ('two', 2, 30),
> ('three', 3, 30),
> ('four', 4, 60),
> ('five', 5, 70),
> ('six', 6, 80);
> 2. Try full outer with the below command. The result is incomplete, it is
> missing the row:
> NULL NULL NULL three 3 30.0
> Full Outer Join:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 full outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> --+
> x1.z x1.x x1.y x2.z x2.x x2.y
> --+
> one 1 50 NULL NULL NULL
> NULL NULL NULL one 1 50
> two 2 30 NULL NULL NULL
> NULL NULL NULL two 2 30
> three 3 30 NULL NULL NULL
> four 4 60 NULL NULL NULL
> NULL NULL NULL four 4 60
> five 5 70 NULL NULL NULL
> NULL NULL NULL five 5 70
> six 6 80 NULL NULL NULL
> NULL NULL NULL six 6 80
> --+
> 3. Expected output is coming when we use left/right join + union:
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 left outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`)
> union
> select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
> x2.`x`, x2.`y`
> from `x` x1 right outer join
> `x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
> x2.`x`);
> Result:
> +
> z x y _col3 _col4 _col5
> +
> NULL NULL NULL five 5 70
> NULL NULL NULL four 4 60
> NULL NULL NULL one 1 50
> four 4 60 NULL NULL NULL
> one 1 50 NULL NULL NULL
> six 6 80 NULL NULL NULL
> three 3 30 NULL NULL NULL
> two 2 30 NULL NULL NULL
> NULL NULL NULL six 6 80
> NULL NULL NULL three 3 30
> NULL NULL NULL two 2 30
> five 5 70 NULL NULL NULL
> +
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Open (was: Patch Available) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch, > HIVE-23347.6.patch, HIVE-23347.7.patch, HIVE-23347.8.patch, HIVE-23347.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Patch Available (was: Open) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch, > HIVE-23347.6.patch, HIVE-23347.7.patch, HIVE-23347.8.patch, HIVE-23347.9.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118313#comment-17118313
]
Hive QA commented on HIVE-23530:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004177/HIVE-23530.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 17287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_infinity]
(batchId=108)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hll]
(batchId=120)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=79)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=45)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=32)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=58)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=42)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part]
(batchId=58)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_llap_io]
(batchId=116)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_part_llap_io]
(batchId=33)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part]
(batchId=107)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive]
(batchId=121)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_all_primitive_llap_io]
(batchId=118)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_part_llap_io]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_table]
(batchId=41)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_nonvec_table_llap_io]
(batchId=113)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part_all_primitive]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part_all_primitive_llap_io]
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_table]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_table_llap_io]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_all_primitive]
(batchId=81)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_all_primitive_llap_io]
(batchId=121)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_part_llap_io]
(batchId=60)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_table]
(batchId=44)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_nonvec_table_llap_io]
(batchId=103)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive]
(batchId=92)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_primitive_llap_io]
(batchId=108)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_llap_io]
(batchId=120)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table_llap_io]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part]
(batchId=115)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive]
(batchId=93)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_primitive_llap_io]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_llap_io]
(batchId=80)
[jira] [Commented] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118302#comment-17118302
]
Hive QA commented on HIVE-23530:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} accumulo-handler in master has 20 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} contrib in master has 11 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} ql: The patch generated 18 new + 144 unchanged - 59
fixed = 162 total (was 203) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22659/dev-support/hive-personality.sh
|
| git revision | master / 9ec54d5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22659/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql accumulo-handler contrib hbase-handler itests/hive-blobstore
U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22659/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.04.patch, HIVE-23530.patch
>
> Time Spent: 1h 40m
> Remaining
[jira] [Updated] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23453: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23453.02.patch, HIVE-23453.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?focusedWorklogId=438165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438165 ] ASF GitHub Bot logged work on HIVE-23453: - Author: ASF GitHub Bot Created on: 28/May/20 04:19 Start Date: 28/May/20 04:19 Worklog Time Spent: 10m Work Description: asfgit closed pull request #1037: URL: https://github.com/apache/hive/pull/1037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 438165) Time Spent: 20m (was: 10m) > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23453.02.patch, HIVE-23453.patch > > Time Spent: 20m > Remaining Estimate: 0h > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118263#comment-17118263
]
Hive QA commented on HIVE-23462:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004170/HIVE-23462.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17291 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing_2]
(batchId=57)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22658/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22658/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22658/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004170 - PreCommit-HIVE-Build
> Add option to rewrite NTILE to sketch functions
> ---
>
> Key: HIVE-23462
> URL: https://issues.apache.org/jira/browse/HIVE-23462
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch,
> HIVE-23462.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23519) Read Ranger Configs from Classpath
[ https://issues.apache.org/jira/browse/HIVE-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anishek Agarwal updated HIVE-23519: --- Resolution: Fixed Status: Resolved (was: Patch Available) +1 , Merged to master, Thanks for the patch [~aasha] and review [~pkumarsinha] > Read Ranger Configs from Classpath > -- > > Key: HIVE-23519 > URL: https://issues.apache.org/jira/browse/HIVE-23519 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23519.01.patch, HIVE-23519.02.patch, > HIVE-23519.03.patch, HIVE-23519.04.patch, HIVE-23519.05.patch, > HIVE-23519.06.patch, HIVE-23519.08.patch, HIVE-23519.09.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23461) Needs to capture input/output entities in explainRewrite
[
https://issues.apache.org/jira/browse/HIVE-23461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naresh P R updated HIVE-23461:
--
Attachment: HIVE-23461.2.patch
> Needs to capture input/output entities in explainRewrite
>
>
> Key: HIVE-23461
> URL: https://issues.apache.org/jira/browse/HIVE-23461
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Reporter: Wenchao Li
>Assignee: Naresh P R
>Priority: Major
> Attachments: HIVE-23461.1.patch, HIVE-23461.2.patch, HIVE-23461.patch
>
>
> HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
> analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
> authorizizer, the corresponding explain statement will be disallowed either.
> However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
> DDLSemanticAnalyzer to analyze the explain rewrite query.
> {code:java}
> SemanticAnalyzer sem = (SemanticAnalyzer)
> SemanticAnalyzerFactory.get(queryState, input);
> sem.analyze(input, ctx);
> sem.validate();{code}
>
> The inputs/outputs entities for this query are never set on the instance of
> ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
> HookContext in the calling Driver code. It is a similar issue to HIVE-18778.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Attachment: HIVE-23526.3.patch > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.3.patch, > HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118253#comment-17118253
]
Hive QA commented on HIVE-23462:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
59s{color} | {color:red} ql: The patch generated 17 new + 180 unchanged - 2
fixed = 197 total (was 182) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
35s{color} | {color:red} ql generated 1 new + 1524 unchanged - 0 fixed = 1525
total (was 1524) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Return value of org.apache.calcite.rex.RexCall.getOperands() ignored, but
method has no side effect At HiveRewriteToDataSketchesRules.java:but method
has no side effect At HiveRewriteToDataSketchesRules.java:[line 498] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22658/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22658/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22658/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22658/yetus/new-findbugs-ql.html
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22658/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add option to rewrite NTILE to sketch functions
> ---
>
> Key: HIVE-23462
> URL: https://issues.apache.org/jira/browse/HIVE-23462
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch,
>
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Issue Type: Bug (was: Improvement) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned HIVE-23526: -- Assignee: (was: Zhihua Deng) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Bug > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Comment: was deleted (was: [~pvary], [~ashutoshc], [~anishek] could you take a look? thanks.) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Description: Sometimes we can see 'out of sequence response' message in beeline, for example: Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of sequence response (state=08S01,code=0) java.sql.SQLException: org.apache.thrift.TApplicationException: CloseOperation failed: out of sequence response at org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) at org.apache.hive.beeline.Commands.execute(Commands.java:891) at org.apache.hive.beeline.Commands.sql(Commands.java:713) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) and there is no other usage messages to figured it out, even with --verbose, this makes problem puzzled as beeline does not have concurrency problem on underlying thrift transport. was: Sometimes we can see 'out of sequence response' message in beeline, for example: Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of sequence response (state=08S01,code=0) java.sql.SQLException: org.apache.thrift.TApplicationException: CloseOperation failed: out of sequence response at org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) at org.apache.hive.beeline.Commands.execute(Commands.java:891) at org.apache.hive.beeline.Commands.sql(Commands.java:713) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) and there is no other usage messages to figured it out, even with verbose flag, this makes problem puzzled as beeline does not have concurrency problem on underlying thrift transport. > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with --verbose, > this makes problem puzzled as beeline does not have concurrency problem on > underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23526) Beeline may throw the misleading exception
[ https://issues.apache.org/jira/browse/HIVE-23526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated HIVE-23526: --- Summary: Beeline may throw the misleading exception (was: Out of sequence seen in Beeline may swallow the real problem ) > Beeline may throw the misleading exception > -- > > Key: HIVE-23526 > URL: https://issues.apache.org/jira/browse/HIVE-23526 > Project: Hive > Issue Type: Improvement > Components: Beeline > Environment: Hive 1.2.2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23526.2.patch, HIVE-23526.patch, outofsequence.log > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes we can see 'out of sequence response' message in beeline, for > example: > Error: org.apache.thrift.TApplicationException: CloseOperation failed: out of > sequence response (state=08S01,code=0) > java.sql.SQLException: org.apache.thrift.TApplicationException: > CloseOperation failed: out of sequence response > at > org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:198) > at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:217) > at org.apache.hive.beeline.Commands.execute(Commands.java:891) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:976) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:816) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:774) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:487) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:470) > and there is no other usage messages to figured it out, even with verbose > flag, this makes problem puzzled as beeline does not have concurrency problem > on underlying thrift transport. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[
https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118238#comment-17118238
]
Hive QA commented on HIVE-23555:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004168/HIVE-23555.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 17079 tests
executed
*Failed tests:*
{noformat}
TestBasicStats - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestBytesBytesMultiHashMap - did not produce a TEST-*.xml file (likely timed
out) (batchId=294)
TestConvertAstToSearchArg - did not produce a TEST-*.xml file (likely timed
out) (batchId=258)
TestCustomPartitionVertex - did not produce a TEST-*.xml file (likely timed
out) (batchId=290)
TestDagUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=290)
TestExplainTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestFunctionRegistry - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestHashPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestHiveKVResultCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestHostAffinitySplitLocationProvider - did not produce a TEST-*.xml file
(likely timed out) (batchId=290)
TestLocalHiveSparkClient - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestMapJoinEqualityTableContainer - did not produce a TEST-*.xml file (likely
timed out) (batchId=294)
TestMapJoinKey - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestMapJoinMemoryExhaustionHandler - did not produce a TEST-*.xml file (likely
timed out) (batchId=293)
TestMapJoinTableContainer - did not produce a TEST-*.xml file (likely timed
out) (batchId=294)
TestMapRedTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestMapredLocalTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestMsckCreatePartitionsInBatches - did not produce a TEST-*.xml file (likely
timed out) (batchId=292)
TestPTFRowContainer - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestPartitionKeySampler - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestPlan - did not produce a TEST-*.xml file (likely timed out) (batchId=293)
TestRangerDumpTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestRangerLoadTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestReplDumpTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestSearchArgumentImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestSmallTableCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestSparkInvalidFileFormat - did not produce a TEST-*.xml file (likely timed
out) (batchId=293)
TestSparkJobMonitor - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestSparkPlan - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestSparkSessionManagerImpl - did not produce a TEST-*.xml file (likely timed
out) (batchId=292)
TestSparkTask - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestSparkUtilities - did not produce a TEST-*.xml file (likely timed out)
(batchId=292)
TestStatsReplicationScenariosMMNoAutogather - did not produce a TEST-*.xml file
(likely timed out) (batchId=183)
TestStatsUpdaterThread - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestStatsUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=258)
TestTaskLogProcessor - did not produce a TEST-*.xml file (likely timed out)
(batchId=294)
TestTaskTracker - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestTezTask - did not produce a TEST-*.xml file (likely timed out) (batchId=290)
TestUtilities - did not produce a TEST-*.xml file (likely timed out)
(batchId=293)
TestVectorFilterOperator - did not produce a TEST-*.xml file (likely timed out)
(batchId=290)
TestVectorHashKeyWrapperBatch - did not produce a TEST-*.xml file (likely timed
out) (batchId=290)
TestVectorMapJoinFastBytesHashMultiSet - did not produce a TEST-*.xml file
(likely timed out) (batchId=290)
TestVectorMapJoinFastLongHashMultiSet - did not produce a TEST-*.xml file
(likely timed out) (batchId=290)
TestVectorMapJoinFastLongHashSet - did not produce a TEST-*.xml file (likely
timed out) (batchId=290)
TestVectorRowBytesContainer - did not produce a TEST-*.xml file (likely timed
out) (batchId=290)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=194)
[jira] [Commented] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[
https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118215#comment-17118215
]
Hive QA commented on HIVE-23555:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
58s{color} | {color:red} ql: The patch generated 4 new + 62 unchanged - 3 fixed
= 66 total (was 65) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} ql generated 0 new + 1523 unchanged - 1 fixed = 1523
total (was 1524) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22657/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22657/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22657/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cancel compaction jobs when hive.compactor.worker.timeout is reached
>
>
> Key: HIVE-23555
> URL: https://issues.apache.org/jira/browse/HIVE-23555
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23555.02.patch, HIVE-23555.patch
>
>
> Currently when a compactor worker thread is stuck, or working too long on a
> compaction the the initiator might decide to start a new compaction because
> of a timeout, but old worker might still wait for the results of the job.
> It would be good to cancel the worker as well after the timeout is reached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23559) Optimise Hive::moveAcidFiles for cloud storage
[ https://issues.apache.org/jira/browse/HIVE-23559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23559: Issue Type: Improvement (was: Bug) > Optimise Hive::moveAcidFiles for cloud storage > -- > > Key: HIVE-23559 > URL: https://issues.apache.org/jira/browse/HIVE-23559 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4752] > It ends up transferring DELTA, DELETE_DELTA, BASE prefixes sequentially from > staging to final location. > This causes delays even with simple updates statements, which updates smaller > number of records in cloud storage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Open (was: Patch Available) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch, HIVE-23468.8.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Attachment: HIVE-23468.8.patch > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch, HIVE-23468.8.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Patch Available (was: Open) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch, HIVE-23468.8.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118190#comment-17118190
]
Hive QA commented on HIVE-23468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004165/HIVE-23468.7.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17288 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestAcidTxnCleanerService.cleansAllCommittedTxns
(batchId=154)
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands (batchId=139)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22656/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22656/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22656/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004165 - PreCommit-HIVE-Build
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch,
> HIVE-23468.6.patch, HIVE-23468.7.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118182#comment-17118182
]
Hive QA commented on HIVE-23468:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22656/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22656/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch,
> HIVE-23468.6.patch, HIVE-23468.7.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118173#comment-17118173
]
Hive QA commented on HIVE-23353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
56s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
14s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 13m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22655/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22655/yetus/patch-findbugs-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22655/yetus/patch-findbugs-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22655/yetus/patch-findbugs-itests_hive-unit.txt
|
| modules | C: common ql . itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22655/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
>
[jira] [Commented] (HIVE-23353) Atlas metadata replication scheduling
[
https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118163#comment-17118163
]
Hive QA commented on HIVE-23353:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004162/HIVE-23353.09.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17292 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22655/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22655/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22655/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004162 - PreCommit-HIVE-Build
> Atlas metadata replication scheduling
> -
>
> Key: HIVE-23353
> URL: https://issues.apache.org/jira/browse/HIVE-23353
> Project: Hive
> Issue Type: Task
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch,
> HIVE-23353.03.patch, HIVE-23353.04.patch, HIVE-23353.05.patch,
> HIVE-23353.06.patch, HIVE-23353.07.patch, HIVE-23353.08.patch,
> HIVE-23353.08.patch, HIVE-23353.08.patch, HIVE-23353.08.patch,
> HIVE-23353.09.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118151#comment-17118151
]
Hive QA commented on HIVE-23453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004160/HIVE-23453.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17288 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004160 - PreCommit-HIVE-Build
> IntelliJ compile errors in StaticPermanentFunctionChecker and
> TestVectorGroupByOperator
> ---
>
> Key: HIVE-23453
> URL: https://issues.apache.org/jira/browse/HIVE-23453
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91
> jdk 1.8.0_251
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23453.02.patch, HIVE-23453.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following errors appear when compiling the code using IntelliJ:
> TestVectorGroupByOperator: Error:(89, 32) java: package
> com.sun.tools.javac.util does not exist
> StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does
> not exist
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-23530:
---
Attachment: HIVE-23530.04.patch
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.04.patch, HIVE-23530.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[
https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118142#comment-17118142
]
Hive QA commented on HIVE-23453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
32s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} llap-server: The patch generated 0 new + 3 unchanged
- 4 fixed = 3 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22654/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22654/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> IntelliJ compile errors in StaticPermanentFunctionChecker and
> TestVectorGroupByOperator
> ---
>
> Key: HIVE-23453
> URL: https://issues.apache.org/jira/browse/HIVE-23453
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91
> jdk 1.8.0_251
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23453.02.patch, HIVE-23453.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following errors appear when compiling the code using IntelliJ:
> TestVectorGroupByOperator: Error:(89, 32) java: package
> com.sun.tools.javac.util does not exist
> StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does
> not exist
--
This message was sent by Atlassian
[jira] [Commented] (HIVE-23485) Bound GroupByOperator stats using largest NDV among columns
[
https://issues.apache.org/jira/browse/HIVE-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118106#comment-17118106
]
Hive QA commented on HIVE-23485:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004158/HIVE-23485.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 17288 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=224)
org.apache.hadoop.hive.cli.TestKuduCliDriver.testCliDriver[kudu_complex_queries]
(batchId=223)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_bhif]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=17)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez_tag]
(batchId=16)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_join_part_col_char]
(batchId=16)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query33]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query41]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query45]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query54]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query56]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query58]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query60]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query6]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query83]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query8]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query23]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query33]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query41]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query45]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query54]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query56]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query58]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query60]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query6]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query83]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query8]
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22653/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004158 - PreCommit-HIVE-Build
> Bound GroupByOperator stats using largest NDV among columns
> ---
>
> Key: HIVE-23485
> URL: https://issues.apache.org/jira/browse/HIVE-23485
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23485.01.patch, HIVE-23485.02.patch
>
>
> Consider the following SQL query:
> {code:sql}
> select id, name from person group by id, name;
> {code}
> and assume that the person table contains the following tuples:
> {code:sql}
> insert into person values (0, 'A') ;
> insert into person values (1, 'A') ;
> insert into person values (2, 'B') ;
> insert into person values (3, 'B') ;
> insert into person values (4, 'B') ;
> insert into person values (5, 'C') ;
> {code}
> If we know the number of distinct values (NDV) for all columns in the group
> by clause then we can infer a lower bound for the total number of rows by
> taking the
[jira] [Commented] (HIVE-23365) Put RS deduplication optimization under cost based decision
[
https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118044#comment-17118044
]
Hive QA commented on HIVE-23365:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004153/HIVE-23365.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 17285 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join32]
(batchId=118)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constant_prop_3]
(batchId=74)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query1]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query1b]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query30]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query65]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query81]
(batchId=230)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query1]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query1b]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query30]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query65]
(batchId=229)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query81]
(batchId=229)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22652/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22652/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22652/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004153 - PreCommit-HIVE-Build
> Put RS deduplication optimization under cost based decision
> ---
>
> Key: HIVE-23365
> URL: https://issues.apache.org/jira/browse/HIVE-23365
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch,
> HIVE-23365.03.patch, HIVE-23365.04.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, RS deduplication is always executed whenever it is semantically
> correct. However, it could be beneficial to leave both RS operators in the
> plan, e.g., if the NDV of the second RS is very low. Thus, we would like this
> decision to be cost-based. We could use a simple heuristic that would work
> fine for most of the cases without introducing regressions for existing
> cases, e.g., if NDV for partition column is less than estimated parallelism
> in the second RS, do not execute deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23365) Put RS deduplication optimization under cost based decision
[
https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118030#comment-17118030
]
Hive QA commented on HIVE-23365:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22652/dev-support/hive-personality.sh
|
| git revision | master / 4d16b4d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22652/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Put RS deduplication optimization under cost based decision
> ---
>
> Key: HIVE-23365
> URL: https://issues.apache.org/jira/browse/HIVE-23365
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch,
> HIVE-23365.03.patch, HIVE-23365.04.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, RS deduplication is always executed whenever it is semantically
> correct. However, it could be beneficial to leave both RS operators in the
> plan, e.g., if the NDV of the second RS is very low. Thus, we would like this
> decision to be cost-based. We could use a simple heuristic that would work
> fine for most of the cases without introducing regressions for existing
> cases, e.g., if NDV for partition column is less than estimated parallelism
> in the second RS, do not execute deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23404) Schedules in the past should be accepted
[ https://issues.apache.org/jira/browse/HIVE-23404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23404: --- Assignee: Zoltan Haindrich > Schedules in the past should be accepted > > > Key: HIVE-23404 > URL: https://issues.apache.org/jira/browse/HIVE-23404 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > Right now if a schedule in the past is set ; an Exception is thrown. > However this behaviour can be argued...consider that at some point a schedule > is created to run in say every day in 2020's december(but not after it). When > the creation happens it's "okay"...but in case we reach 2021 it will be > considered invalid..because there is no future execution -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118020#comment-17118020
]
Hive QA commented on HIVE-23462:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
31s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
58s{color} | {color:red} ql: The patch generated 16 new + 180 unchanged - 2
fixed = 196 total (was 182) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
9s{color} | {color:red} root: The patch generated 16 new + 557 unchanged - 2
fixed = 573 total (was 559) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m
25s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 13m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22650/dev-support/hive-personality.sh
|
| git revision | master / f49d257 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus/patch-findbugs-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus/patch-findbugs-ql.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22650/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add option to rewrite NTILE to sketch functions
> ---
>
> Key: HIVE-23462
> URL: https://issues.apache.org/jira/browse/HIVE-23462
>
[jira] [Commented] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[
https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118016#comment-17118016
]
Hive QA commented on HIVE-23556:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004150/HIVE-23556.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22651/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22651/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22651/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-27 18:40:28.224
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22651/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-27 18:40:28.252
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
f49d257..a3a25eb master -> origin/master
+ git reset --hard HEAD
HEAD is now at f49d257 HIVE-23547 Enforce testconfiguration.properties file
format and alphabetical order (Miklos Gergely, reviewed by Laszlo Bodor)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at a3a25eb HIVE-23488 : Optimise
PartitionManagementTask::Msck::repair (Rajesh Balamohan via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-27 18:40:40.790
+ rm -rf ../yetus_PreCommit-HIVE-Build-22651
+ mkdir ../yetus_PreCommit-HIVE-Build-22651
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22651
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22651/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6550287692100694868.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc6550287692100694868.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc8384485657991550447.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
[jira] [Updated] (HIVE-23531) Major CRUD QB compaction failing with ClassCastException when vectorization off
[
https://issues.apache.org/jira/browse/HIVE-23531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23531:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~klcopp], and [~lpinter] for the review!
> Major CRUD QB compaction failing with ClassCastException when vectorization
> off
> ---
>
> Key: HIVE-23531
> URL: https://issues.apache.org/jira/browse/HIVE-23531
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23531.01.patch, HIVE-23531.01.patch,
> HIVE-23531.01.patch, HIVE-23531.01.patch
>
>
> Exception:
> {code:java}
> 2020-05-22T01:33:09,944 ERROR [TezChild] tez.MapRecordSource:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:403)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.IntWritable
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:965)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:887)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:887)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:174)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 20 more
> {code}
> And some more in Tez.
> Because when vectorization is turned on, primitives in the row are wrapped in
> Writables by VectorFileSinkOperator; when it is off, they are not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23462: Attachment: HIVE-23462.03.patch > Add option to rewrite NTILE to sketch functions > --- > > Key: HIVE-23462 > URL: https://issues.apache.org/jira/browse/HIVE-23462 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch, > HIVE-23462.03.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23492) Remove unnecessary FileSystem#exists calls from ql module
[ https://issues.apache.org/jira/browse/HIVE-23492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23492: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~klcopp]! > Remove unnecessary FileSystem#exists calls from ql module > - > > Key: HIVE-23492 > URL: https://issues.apache.org/jira/browse/HIVE-23492 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23492.01.patch, HIVE-23492.02.patch, > HIVE-23492.03.patch, HIVE-23492.04.patch, HIVE-23492.05.patch > > > Wherever there is an exists() call before open() or delete(), remove it and > infer from the FileNotFoundException raised in open/delete that the file does > not exist. Exists() just checks for a FileNotFoundException so it's a waste > of time, especially on clunkier FSes -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118002#comment-17118002
]
Hive QA commented on HIVE-23462:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004137/HIVE-23462.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 330 failed/errored test(s), 17289 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input39] (batchId=4)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=130)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=130)
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_basic]
(batchId=227)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_subquery]
(batchId=70)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_view_delete]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_windowing_2]
(batchId=57)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_stats_estimation]
(batchId=116)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_exists]
(batchId=109)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_in]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constraints_optimization]
(batchId=79)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[create_transactional_full_acid]
(batchId=107)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_join]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_2]
(batchId=108)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explain_locks]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=64)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[external_jdbc_table_perf]
(batchId=115)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[get_splits_0]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[input41]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[intersect_all_rj]
(batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join46]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_reorder5]
(batchId=52)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_reordering_no_stats]
(batchId=94)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb_ptf]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[masking_12]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[masking_3]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[masking_4]
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned_2]
(batchId=93)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_2]
(batchId=52)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_4]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_6]
(batchId=101)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_part_2]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb]
(batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_ssb_2]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_window]
(batchId=107)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
(batchId=98)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[nested_column_pruning]
(batchId=63)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_nested_column_pruning]
(batchId=81)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin4]
(batchId=119)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin5]
(batchId=44)
[jira] [Commented] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[ https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118000#comment-17118000 ] Peter Vary commented on HIVE-23555: --- RB link: https://reviews.apache.org/r/72553/ > Cancel compaction jobs when hive.compactor.worker.timeout is reached > > > Key: HIVE-23555 > URL: https://issues.apache.org/jira/browse/HIVE-23555 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23555.02.patch, HIVE-23555.patch > > > Currently when a compactor worker thread is stuck, or working too long on a > compaction the the initiator might decide to start a new compaction because > of a timeout, but old worker might still wait for the results of the job. > It would be good to cancel the worker as well after the timeout is reached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[ https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23555: -- Attachment: HIVE-23555.02.patch > Cancel compaction jobs when hive.compactor.worker.timeout is reached > > > Key: HIVE-23555 > URL: https://issues.apache.org/jira/browse/HIVE-23555 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23555.02.patch, HIVE-23555.patch > > > Currently when a compactor worker thread is stuck, or working too long on a > compaction the the initiator might decide to start a new compaction because > of a timeout, but old worker might still wait for the results of the job. > It would be good to cancel the worker as well after the timeout is reached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437947
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 18:21
Start Date: 27/May/20 18:21
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431351279
##
File path:
accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
##
@@ -96,16 +96,16 @@ STAGE PLANS:
outputColumnNames: key, value
Statistics: Num rows: 55 Data size: 9405 Basic stats:
COMPLETE Column stats: COMPLETE
Group By Operator
-aggregations: compute_stats(key, 'hll'),
compute_stats(value, 'hll')
+aggregations: max(length(key)),
avg(COALESCE(length(key),0)), count(CASE WHEN (key is null) THEN (1) ELSE
(null) END), compute_bit_vector(key, 'hll'), max(length(value)),
avg(COALESCE(length(value),0)), count(CASE WHEN (value is null) THEN (1) ELSE
(null) END), compute_bit_vector(value, 'hll')
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437947)
Time Spent: 1h 40m (was: 1.5h)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117985#comment-17117985
]
Zoltan Haindrich commented on HIVE-23530:
-
+1
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[ https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23488: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Rajesh! > Optimise PartitionManagementTask::Msck::repair > -- > > Key: HIVE-23488 > URL: https://issues.apache.org/jira/browse/HIVE-23488 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23488.1.patch, HIVE-23488.2.patch, > HIVE-23488.3.patch, Screenshot 2020-05-18 at 5.06.15 AM.png > > > Ends up fetching table information twice. > !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754! > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113] > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-19926) Remove deprecated hcatalog streaming
[ https://issues.apache.org/jira/browse/HIVE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117973#comment-17117973 ] Zoltan Haindrich commented on HIVE-19926: - [~ashutoshc] yes it seems like it somehow expects hcatalog/streaming... taking a quick look at the conf file I don't see anything outstanding... since HIVE-22942 is close to be operation - I would like to opt to merge that - and run thiese changes thru the new system (which will most probably just execute the tests...) http://104.198.109.242/logs/PreCommit-HIVE-Build-22643/failed/151_UTBatch_hcatalog__streaming_16_tests/151_UTBatch_hcatalog__streaming_16_tests.txt > Remove deprecated hcatalog streaming > > > Key: HIVE-19926 > URL: https://issues.apache.org/jira/browse/HIVE-19926 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19926.1.patch, HIVE-19926.2.patch, > HIVE-19926.3.patch, HIVE-19926.4.patch, HIVE-19926.5.patch, HIVE-19926.6.patch > > > hcatalog streaming is deprecated in 3.0.0. We should remove it in 4.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Attachment: HIVE-23468.7.patch > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Patch Available (was: Open) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[ https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23468: Status: Open (was: Patch Available) > LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN > -- > > Key: HIVE-23468 > URL: https://issues.apache.org/jira/browse/HIVE-23468 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch, > HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, > HIVE-23468.6.patch, HIVE-23468.7.patch > > > OrcEncodedDataReader materializes the supplier to check if it is a HDFS > system or not. This causes unwanted call to NN even in cases when cache is > completely warmed up. > [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540] > [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107] > Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case. > IO elevator could get 100% cache hit from FileSystem impl in warmed up > scenario. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19926) Remove deprecated hcatalog streaming
[ https://issues.apache.org/jira/browse/HIVE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19926: Attachment: HIVE-19926.6.patch > Remove deprecated hcatalog streaming > > > Key: HIVE-19926 > URL: https://issues.apache.org/jira/browse/HIVE-19926 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19926.1.patch, HIVE-19926.2.patch, > HIVE-19926.3.patch, HIVE-19926.4.patch, HIVE-19926.5.patch, HIVE-19926.6.patch > > > hcatalog streaming is deprecated in 3.0.0. We should remove it in 4.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19926) Remove deprecated hcatalog streaming
[ https://issues.apache.org/jira/browse/HIVE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19926: Status: Open (was: Patch Available) > Remove deprecated hcatalog streaming > > > Key: HIVE-19926 > URL: https://issues.apache.org/jira/browse/HIVE-19926 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19926.1.patch, HIVE-19926.2.patch, > HIVE-19926.3.patch, HIVE-19926.4.patch, HIVE-19926.5.patch > > > hcatalog streaming is deprecated in 3.0.0. We should remove it in 4.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23353) Atlas metadata replication scheduling
[ https://issues.apache.org/jira/browse/HIVE-23353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha updated HIVE-23353: Attachment: HIVE-23353.09.patch > Atlas metadata replication scheduling > - > > Key: HIVE-23353 > URL: https://issues.apache.org/jira/browse/HIVE-23353 > Project: Hive > Issue Type: Task >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23353.01.patch, HIVE-23353.02.patch, > HIVE-23353.03.patch, HIVE-23353.04.patch, HIVE-23353.05.patch, > HIVE-23353.06.patch, HIVE-23353.07.patch, HIVE-23353.08.patch, > HIVE-23353.08.patch, HIVE-23353.08.patch, HIVE-23353.08.patch, > HIVE-23353.09.patch > > Time Spent: 4h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117953#comment-17117953 ] Zoltan Haindrich commented on HIVE-22942: - [~jcamachorodriguez]: could you please take a look? > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22942.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework > some info/etc about how it compares to existing one: > https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-19926) Remove deprecated hcatalog streaming
[ https://issues.apache.org/jira/browse/HIVE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117951#comment-17117951 ] Ashutosh Chauhan commented on HIVE-19926: - Seems like we need to disable HCatalog streaming tests from being executed. Tests themselves are deleted as part of this patch. I am assuming test batch creation logic somehow still tries to execute these tests. Do we need to modify test servers config to prevent these tests from being executed? > Remove deprecated hcatalog streaming > > > Key: HIVE-19926 > URL: https://issues.apache.org/jira/browse/HIVE-19926 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19926.1.patch, HIVE-19926.2.patch, > HIVE-19926.3.patch, HIVE-19926.4.patch, HIVE-19926.5.patch > > > hcatalog streaming is deprecated in 3.0.0. We should remove it in 4.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-19926) Remove deprecated hcatalog streaming
[ https://issues.apache.org/jira/browse/HIVE-19926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117951#comment-17117951 ] Ashutosh Chauhan edited comment on HIVE-19926 at 5/27/20, 5:48 PM: --- Seems like we need to disable HCatalog streaming tests from being executed. Tests themselves are deleted as part of this patch. I am assuming test batch creation logic somehow still tries to execute these tests. Do we need to modify test servers config to prevent these tests from being executed? [~kgyrtkirk] ? was (Author: ashutoshc): Seems like we need to disable HCatalog streaming tests from being executed. Tests themselves are deleted as part of this patch. I am assuming test batch creation logic somehow still tries to execute these tests. Do we need to modify test servers config to prevent these tests from being executed? > Remove deprecated hcatalog streaming > > > Key: HIVE-19926 > URL: https://issues.apache.org/jira/browse/HIVE-19926 > Project: Hive > Issue Type: Improvement > Components: Streaming >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19926.1.patch, HIVE-19926.2.patch, > HIVE-19926.3.patch, HIVE-19926.4.patch, HIVE-19926.5.patch > > > hcatalog streaming is deprecated in 3.0.0. We should remove it in 4.0.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[
https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117940#comment-17117940
]
Hive QA commented on HIVE-23555:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004132/HIVE-23555.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17286 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=76)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22649/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22649/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22649/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004132 - PreCommit-HIVE-Build
> Cancel compaction jobs when hive.compactor.worker.timeout is reached
>
>
> Key: HIVE-23555
> URL: https://issues.apache.org/jira/browse/HIVE-23555
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23555.patch
>
>
> Currently when a compactor worker thread is stuck, or working too long on a
> compaction the the initiator might decide to start a new compaction because
> of a timeout, but old worker might still wait for the results of the job.
> It would be good to cancel the worker as well after the timeout is reached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23365) Put RS deduplication optimization under cost based decision
[
https://issues.apache.org/jira/browse/HIVE-23365?focusedWorklogId=437920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437920
]
ASF GitHub Bot logged work on HIVE-23365:
-
Author: ASF GitHub Bot
Created on: 27/May/20 17:18
Start Date: 27/May/20 17:18
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1035:
URL: https://github.com/apache/hive/pull/1035#discussion_r431276400
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
##
@@ -113,13 +116,36 @@ public static boolean merge(ReduceSinkOperator cRS,
JoinOperator pJoin, int minR
* If parent RS has not been assigned any partitioning column, we will use
* partitioning columns (if exist) of child RS.
*/
- public static boolean merge(ReduceSinkOperator cRS, ReduceSinkOperator pRS,
int minReducer)
+ public static boolean merge(HiveConf hiveConf, ReduceSinkOperator cRS,
ReduceSinkOperator pRS, int minReducer)
throws SemanticException {
int[] result = extractMergeDirections(cRS, pRS, minReducer);
if (result == null) {
return false;
}
+// The partitioning columns of the child RS will replace the columns of the
+// parent RS in two cases:
+// - Parent RS columns are more specific than those of the child RS,
+// and child columns are assigned;
+// - Child RS columns are more specific than those of the parent RS,
+// and parent columns are not assigned.
+List childPCs = cRS.getConf().getPartitionCols();
+List parentPCs = pRS.getConf().getPartitionCols();
+boolean useChildsPartitionColumns =
+result[1] < 0 && (childPCs != null && !childPCs.isEmpty()) ||
+result[1] > 0 && (parentPCs == null || parentPCs.isEmpty());
+
+if (useChildsPartitionColumns) {
+ List newPartitionCols =
ExprNodeDescUtils.backtrack(childPCs, cRS, pRS);
+ long oldParallelism = estimateMaxPartitions(hiveConf, pRS, parentPCs);
+ long newParallelism = estimateMaxPartitions(hiveConf, pRS,
newPartitionCols);
+ long threshold =
hiveConf.getLongVar(HiveConf.ConfVars.HIVEOPTREDUCEDEDUPLICATIONPARALLELISMDECTHRESHOLD);
+ if (oldParallelism / newParallelism > threshold) {
+return false;
+ }
Review comment:
I think you are right, adding the check using the existing config seems
to be the correct approach. We could still add on/off config for the new
behavior optimization (default true... but in case we need to disable it).
Could you make those changes?
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
##
@@ -113,13 +116,36 @@ public static boolean merge(ReduceSinkOperator cRS,
JoinOperator pJoin, int minR
* If parent RS has not been assigned any partitioning column, we will use
* partitioning columns (if exist) of child RS.
*/
- public static boolean merge(ReduceSinkOperator cRS, ReduceSinkOperator pRS,
int minReducer)
+ public static boolean merge(HiveConf hiveConf, ReduceSinkOperator cRS,
ReduceSinkOperator pRS, int minReducer)
throws SemanticException {
int[] result = extractMergeDirections(cRS, pRS, minReducer);
if (result == null) {
return false;
}
+// The partitioning columns of the child RS will replace the columns of the
+// parent RS in two cases:
+// - Parent RS columns are more specific than those of the child RS,
+// and child columns are assigned;
+// - Child RS columns are more specific than those of the parent RS,
+// and parent columns are not assigned.
+List childPCs = cRS.getConf().getPartitionCols();
+List parentPCs = pRS.getConf().getPartitionCols();
+boolean useChildsPartitionColumns =
+result[1] < 0 && (childPCs != null && !childPCs.isEmpty()) ||
+result[1] > 0 && (parentPCs == null || parentPCs.isEmpty());
+
+if (useChildsPartitionColumns) {
+ List newPartitionCols =
ExprNodeDescUtils.backtrack(childPCs, cRS, pRS);
+ long oldParallelism = estimateMaxPartitions(hiveConf, pRS, parentPCs);
+ long newParallelism = estimateMaxPartitions(hiveConf, pRS,
newPartitionCols);
+ long threshold =
hiveConf.getLongVar(HiveConf.ConfVars.HIVEOPTREDUCEDEDUPLICATIONPARALLELISMDECTHRESHOLD);
+ if (oldParallelism / newParallelism > threshold) {
+return false;
Review comment:
Do you think it makes sense to add these checks to the
`extractMergeDirections` method? It seems the rest of checks are done within
that method; if `extractMergeDirections` was successful, this method was only
modifying the operators accordingly. I think keeping that separation may make
the code more clear.
This is an automated message from the
[jira] [Updated] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-23453: --- Attachment: HIVE-23453.02.patch > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23453.02.patch, HIVE-23453.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23453: -- Labels: pull-request-available (was: ) > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23453.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23453) IntelliJ compile errors in StaticPermanentFunctionChecker and TestVectorGroupByOperator
[ https://issues.apache.org/jira/browse/HIVE-23453?focusedWorklogId=437912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437912 ] ASF GitHub Bot logged work on HIVE-23453: - Author: ASF GitHub Bot Created on: 27/May/20 17:03 Start Date: 27/May/20 17:03 Worklog Time Spent: 10m Work Description: zabetak opened a new pull request #1037: URL: https://github.com/apache/hive/pull/1037 1. Replace com.sun.tools.javac.util.Pair with org.apache.calcite.util.Pair in TestVectorGroupByOperator. 2. Remove unused imports in StaticPermanentFunctionChecker in particular com.sun.jdi.InvocationException. Both problems rise from the fact that tools.jar is not in the classpath (compile dependency) and is certainly not worth adding. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 437912) Remaining Estimate: 0h Time Spent: 10m > IntelliJ compile errors in StaticPermanentFunctionChecker and > TestVectorGroupByOperator > --- > > Key: HIVE-23453 > URL: https://issues.apache.org/jira/browse/HIVE-23453 > Project: Hive > Issue Type: Bug > Components: Hive > Environment: IntelliJ IDEA 2020.1.1 built 201.7223.91 > jdk 1.8.0_251 >Reporter: Stamatis Zampetakis >Assignee: Stamatis Zampetakis >Priority: Major > Attachments: HIVE-23453.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The following errors appear when compiling the code using IntelliJ: > TestVectorGroupByOperator: Error:(89, 32) java: package > com.sun.tools.javac.util does not exist > StaticPermanentFunctionChecker: Error:(31, 19) java: package com.sun.jdi does > not exist -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[
https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117920#comment-17117920
]
Hive QA commented on HIVE-23555:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} ql: The patch generated 4 new + 13 unchanged - 3 fixed
= 17 total (was 16) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} ql generated 0 new + 1523 unchanged - 1 fixed = 1523
total (was 1524) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22649/dev-support/hive-personality.sh
|
| git revision | master / f49d257 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22649/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22649/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cancel compaction jobs when hive.compactor.worker.timeout is reached
>
>
> Key: HIVE-23555
> URL: https://issues.apache.org/jira/browse/HIVE-23555
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23555.patch
>
>
> Currently when a compactor worker thread is stuck, or working too long on a
> compaction the the initiator might decide to start a new compaction because
> of a timeout, but old worker might still wait for the results of the job.
> It would be good to cancel the worker as well after the timeout is reached
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117890#comment-17117890
]
Hive QA commented on HIVE-23468:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004125/HIVE-23468.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17286 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=217)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=217)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=217)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=217)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=217)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=217)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22648/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22648/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22648/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004125 - PreCommit-HIVE-Build
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, HIVE-23468.6.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23533) Remove an FS#exists call from AcidUtils#getLogicalLength
[
https://issues.apache.org/jira/browse/HIVE-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-23533:
-
Status: Open (was: Patch Available)
> Remove an FS#exists call from AcidUtils#getLogicalLength
>
>
> Key: HIVE-23533
> URL: https://issues.apache.org/jira/browse/HIVE-23533
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23533.01.patch, HIVE-23533.01.patch,
> HIVE-23533.01.patch
>
>
> {code:java}
> Path lengths = OrcAcidUtils.getSideFile(file.getPath());
> if(!fs.exists(lengths)) {
> ...
> return file.getLen();
> }
> long len = OrcAcidUtils.getLastFlushLength(fs, file.getPath());
> {code}
> OrcAcidUtils.getLastFlushLength also has an exists() check and returns
> Long.MAX_VALUE if false.
> exists() is expensive on S3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23485) Bound GroupByOperator stats using largest NDV among columns
[
https://issues.apache.org/jira/browse/HIVE-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-23485:
---
Attachment: HIVE-23485.02.patch
> Bound GroupByOperator stats using largest NDV among columns
> ---
>
> Key: HIVE-23485
> URL: https://issues.apache.org/jira/browse/HIVE-23485
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: HIVE-23485.01.patch, HIVE-23485.02.patch
>
>
> Consider the following SQL query:
> {code:sql}
> select id, name from person group by id, name;
> {code}
> and assume that the person table contains the following tuples:
> {code:sql}
> insert into person values (0, 'A') ;
> insert into person values (1, 'A') ;
> insert into person values (2, 'B') ;
> insert into person values (3, 'B') ;
> insert into person values (4, 'B') ;
> insert into person values (5, 'C') ;
> {code}
> If we know the number of distinct values (NDV) for all columns in the group
> by clause then we can infer a lower bound for the total number of rows by
> taking the maximun NDV of the involved columns.
> Currently the query in the scenario above has the following plan:
> {noformat}
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2 vectorized
> File Output Operator [FS_11]
> Group By Operator [GBY_10] (rows=3 width=92)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE] vectorized
> SHUFFLE [RS_9]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_8] (rows=3 width=92)
> Output:["_col0","_col1"],keys:id, name
> Select Operator [SEL_7] (rows=6 width=92)
> Output:["id","name"]
> TableScan [TS_0] (rows=6 width=92)
>
> default@person,person,Tbl:COMPLETE,Col:COMPLETE,Output:["id","name"]{noformat}
> Observe that the stats for group by report 3 rows but given that the ID
> attribute is part of the aggregation the rows cannot be less than 6.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
[
https://issues.apache.org/jira/browse/HIVE-23468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117874#comment-17117874
]
Hive QA commented on HIVE-23468:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
25s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22648/dev-support/hive-personality.sh
|
| git revision | master / f49d257 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22648/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
> --
>
> Key: HIVE-23468
> URL: https://issues.apache.org/jira/browse/HIVE-23468
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23468.1.patch, HIVE-23468.2.patch,
> HIVE-23468.3.patch, HIVE-23468.4.patch, HIVE-23468.5.patch, HIVE-23468.6.patch
>
>
> OrcEncodedDataReader materializes the supplier to check if it is a HDFS
> system or not. This causes unwanted call to NN even in cases when cache is
> completely warmed up.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]
> [https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]
> Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.
> IO elevator could get 100% cache hit from FileSystem impl in warmed up
> scenario.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23365) Put RS deduplication optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-23365: --- Attachment: HIVE-23365.04.patch > Put RS deduplication optimization under cost based decision > --- > > Key: HIVE-23365 > URL: https://issues.apache.org/jira/browse/HIVE-23365 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Stamatis Zampetakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch, > HIVE-23365.03.patch, HIVE-23365.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Currently, RS deduplication is always executed whenever it is semantically > correct. However, it could be beneficial to leave both RS operators in the > plan, e.g., if the NDV of the second RS is very low. Thus, we would like this > decision to be cost-based. We could use a simple heuristic that would work > fine for most of the cases without introducing regressions for existing > cases, e.g., if NDV for partition column is less than estimated parallelism > in the second RS, do not execute deduplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437838
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 15:09
Start Date: 27/May/20 15:09
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431216259
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
##
@@ -304,15 +287,199 @@ public static String genRewrittenQuery(Table tbl,
List colNames, HiveCon
}
String rewrittenQuery = rewrittenQueryBuilder.toString();
-rewrittenQuery = new VariableSubstitution(new HiveVariableSource() {
- @Override
- public Map getHiveVariable() {
-return SessionState.get().getHiveVariables();
- }
-}).substitute(conf, rewrittenQuery);
+rewrittenQuery = new VariableSubstitution(
+() -> SessionState.get().getHiveVariables()).substitute(conf,
rewrittenQuery);
return rewrittenQuery;
}
+ private static void genComputeStats(StringBuilder rewrittenQueryBuilder,
HiveConf conf,
+ int pos, String columnName, TypeInfo typeInfo) throws SemanticException {
+Preconditions.checkArgument(typeInfo.getCategory() == Category.PRIMITIVE);
+ColumnStatsType columnStatsType =
+ColumnStatsType.getColumnStatsType((PrimitiveTypeInfo) typeInfo);
+// The first column is always the type
+// The rest of columns will depend on the type itself
+int size = columnStatsType.getColumnStats().size() - 1;
+for (int i = 0; i < size; i++) {
+ ColumnStatsField columnStatsField =
columnStatsType.getColumnStats().get(i);
+ appendStatsField(rewrittenQueryBuilder, conf, columnStatsField,
columnStatsType,
+ columnName, pos);
+ rewrittenQueryBuilder.append(", ");
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437838)
Time Spent: 1.5h (was: 1h 20m)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437836
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 15:06
Start Date: 27/May/20 15:06
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431213790
##
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java
##
@@ -127,15 +131,16 @@ public int process(Hive db, Table tbl) throws Exception {
LOG.debug("Because {} is infinite or NaN, we skip stats.",
columnName, e);
}
}
+pos += columnStatsFields.size();
Review comment:
Changed the method to use iterators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437836)
Time Spent: 1h 20m (was: 1h 10m)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23554) [LLAP] support ColumnVectorBatch with FilterContext as part of ReadPipeline
[
https://issues.apache.org/jira/browse/HIVE-23554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117837#comment-17117837
]
Hive QA commented on HIVE-23554:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004121/HIVE-23554.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17286 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22647/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22647/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22647/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004121 - PreCommit-HIVE-Build
> [LLAP] support ColumnVectorBatch with FilterContext as part of ReadPipeline
> ---
>
> Key: HIVE-23554
> URL: https://issues.apache.org/jira/browse/HIVE-23554
> Project: Hive
> Issue Type: Sub-task
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23554.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the readPipeline in LLAP supports consuming ColumnVectorBatches.
> As each batch can be now tied with a Filter (HIVE-22959 HIVE-23215) we
> should update the pipeline to consume BatchWrappers of ColumnVectorBatch and
> a Filter instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[ https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437817=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437817 ] ASF GitHub Bot logged work on HIVE-23530: - Author: ASF GitHub Bot Created on: 27/May/20 14:44 Start Date: 27/May/20 14:44 Worklog Time Spent: 10m Work Description: jcamachor commented on a change in pull request #1034: URL: https://github.com/apache/hive/pull/1034#discussion_r431192485 ## File path: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeBitVector.java ## @@ -0,0 +1,561 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.ql.udf.generic; + +import org.apache.hadoop.hive.common.classification.InterfaceAudience; +import org.apache.hadoop.hive.common.ndv.NumDistinctValueEstimator; +import org.apache.hadoop.hive.common.type.HiveDecimal; +import org.apache.hadoop.hive.ql.exec.Description; +import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException; +import org.apache.hadoop.hive.ql.metadata.HiveException; +import org.apache.hadoop.hive.ql.parse.SemanticException; +import org.apache.hadoop.hive.ql.stats.ColStatsProcessor.ColumnStatsType; +import org.apache.hadoop.hive.ql.util.JavaDataModel; +import org.apache.hadoop.hive.serde2.io.DateWritableV2; +import org.apache.hadoop.hive.serde2.io.TimestampWritableV2; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.DateObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.DoubleObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.HiveDecimalObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.LongObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector; +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; +import org.apache.hadoop.io.BytesWritable; + +import static org.apache.hadoop.hive.common.ndv.NumDistinctValueEstimatorFactory.getEmptyNumDistinctValueEstimator; +import static org.apache.hadoop.hive.common.ndv.NumDistinctValueEstimatorFactory.getNumDistinctValueEstimator; + +/** + * GenericUDAFComputeBitVector. This UDAF replicates part of the functionality + * that was in GenericUDAFComputeStats previously, which is deprecated now. + * In particular, it will compute a bit vector using the algorithm provided Review comment: Updated the javadoc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 437817) Time Spent: 1h 10m (was: 1h) > Use SQL functions instead of compute_stats UDAF to compute column statistics > > > Key: HIVE-23530 > URL: https://issues.apache.org/jira/browse/HIVE-23530 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch, > HIVE-23530.03.patch, HIVE-23530.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently we compute column statistics by relying on the
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437814
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 14:43
Start Date: 27/May/20 14:43
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431191795
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
##
@@ -61,6 +61,7 @@
*/
@Description(name = "compute_stats",
value = "_FUNC_(x) - Returns the statistical summary of a set of
primitive type values.")
+@Deprecated
Review comment:
Some tests use `compute_stats` directly. I have created a different JIRA
https://issues.apache.org/jira/browse/HIVE-23558 to remove the UDAF and
potentially rewriting/removing those tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437814)
Time Spent: 1h (was: 50m)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23558) Remove compute_stats UDAF
[ https://issues.apache.org/jira/browse/HIVE-23558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-23558: -- > Remove compute_stats UDAF > - > > Key: HIVE-23558 > URL: https://issues.apache.org/jira/browse/HIVE-23558 > Project: Hive > Issue Type: Improvement > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HIVE-23530 replaces its usage completely. This issue is to remove it from > Hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437810=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437810
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 14:40
Start Date: 27/May/20 14:40
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431189978
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
##
@@ -304,15 +287,199 @@ public static String genRewrittenQuery(Table tbl,
List colNames, HiveCon
}
String rewrittenQuery = rewrittenQueryBuilder.toString();
-rewrittenQuery = new VariableSubstitution(new HiveVariableSource() {
- @Override
- public Map getHiveVariable() {
-return SessionState.get().getHiveVariables();
- }
-}).substitute(conf, rewrittenQuery);
+rewrittenQuery = new VariableSubstitution(
+() -> SessionState.get().getHiveVariables()).substitute(conf,
rewrittenQuery);
return rewrittenQuery;
}
+ private static void genComputeStats(StringBuilder rewrittenQueryBuilder,
HiveConf conf,
+ int pos, String columnName, TypeInfo typeInfo) throws SemanticException {
+Preconditions.checkArgument(typeInfo.getCategory() == Category.PRIMITIVE);
+ColumnStatsType columnStatsType =
+ColumnStatsType.getColumnStatsType((PrimitiveTypeInfo) typeInfo);
+// The first column is always the type
+// The rest of columns will depend on the type itself
+int size = columnStatsType.getColumnStats().size() - 1;
+for (int i = 0; i < size; i++) {
+ ColumnStatsField columnStatsField =
columnStatsType.getColumnStats().get(i);
+ appendStatsField(rewrittenQueryBuilder, conf, columnStatsField,
columnStatsType,
+ columnName, pos);
+ rewrittenQueryBuilder.append(", ");
+}
+ColumnStatsField columnStatsField =
columnStatsType.getColumnStats().get(size);
+appendStatsField(rewrittenQueryBuilder, conf, columnStatsField,
columnStatsType,
+columnName, pos);
+ }
+
+ private static void appendStatsField(StringBuilder rewrittenQueryBuilder,
HiveConf conf,
+ ColumnStatsField columnStatsField, ColumnStatsType columnStatsType,
+ String columnName, int pos) throws SemanticException {
+switch (columnStatsField) {
+case COLUMN_TYPE:
+ appendColumnType(rewrittenQueryBuilder, conf, columnStatsType, pos);
Review comment:
This is kind of misleading because `column_type` does not always match
the actual column type, e.g., BYTE, SHORT, INT or BIGINT are all mapped to the
same column stats type (LONG). However, I did not want to change the internal
name that was used before in this patch too. I renamed the enum to
`COLUMN_STATS_TYPE` to add some more clarity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437810)
Time Spent: 50m (was: 40m)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
[jira] [Updated] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[ https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiko Uchida updated HIVE-23556: Attachment: HIVE-23556.patch Assignee: Toshihiko Uchida Status: Patch Available (was: Open) > Support hive.metastore.limit.partition.request for get_partitions_ps > > > Key: HIVE-23556 > URL: https://issues.apache.org/jira/browse/HIVE-23556 > Project: Hive > Issue Type: Improvement >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-23556.patch > > > HIVE-13884 added the configuration hive.metastore.limit.partition.request to > limit the number of partitions that can be requested. > Currently, it takes in effect for the following MetaStore APIs > * get_partitions, > * get_partitions_with_auth, > * get_partitions_by_filter, > * get_partitions_spec_by_filter, > * get_partitions_by_expr, > but not for > * get_partitions_ps, > * get_partitions_ps_with_auth. > This issue proposes to apply the configuration also to get_partitions_ps and > get_partitions_ps_with_auth. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437805
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 14:28
Start Date: 27/May/20 14:28
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431176481
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java
##
@@ -130,8 +130,10 @@ public void insertTableValuesAnalyzePipeline() throws
SemanticException {
partSpec.put(partKey, null);
}
}
+List colNames =
Utilities.getColumnNamesFromFieldSchema(tbl.getCols());
+List colTypes = ColumnStatsSemanticAnalyzer.getColumnTypes(tbl,
colNames);
String command = ColumnStatsSemanticAnalyzer.genRewrittenQuery(
-tbl, Utilities.getColumnNamesFromFieldSchema(tbl.getCols()), conf,
partSpec, isPartitionStats, true);
+tbl, colNames, colTypes, conf, partSpec, isPartitionStats, true);
Review comment:
The underlying `genRewrittenQuery` method may work on a subset of
columns in the table, that is why it was receiving column names / types
separately. I have made the `protected static` method work directly with the
table and extract the column names and types from it, and added corresponding
comments to it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437805)
Time Spent: 40m (was: 0.5h)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23554) [LLAP] support ColumnVectorBatch with FilterContext as part of ReadPipeline
[
https://issues.apache.org/jira/browse/HIVE-23554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117806#comment-17117806
]
Hive QA commented on HIVE-23554:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} llap-server in master has 88 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 1 new + 73 unchanged
- 1 fixed = 74 total (was 74) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22647/dev-support/hive-personality.sh
|
| git revision | master / f49d257 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22647/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22647/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> [LLAP] support ColumnVectorBatch with FilterContext as part of ReadPipeline
> ---
>
> Key: HIVE-23554
> URL: https://issues.apache.org/jira/browse/HIVE-23554
> Project: Hive
> Issue Type: Sub-task
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23554.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the readPipeline in LLAP supports consuming ColumnVectorBatches.
> As each batch can be now tied with a Filter (HIVE-22959 HIVE-23215) we
> should update the pipeline to consume BatchWrappers of ColumnVectorBatch and
> a Filter instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23365) Put RS deduplication optimization under cost based decision
[
https://issues.apache.org/jira/browse/HIVE-23365?focusedWorklogId=437802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437802
]
ASF GitHub Bot logged work on HIVE-23365:
-
Author: ASF GitHub Bot
Created on: 27/May/20 14:21
Start Date: 27/May/20 14:21
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1035:
URL: https://github.com/apache/hive/pull/1035#discussion_r431170080
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
##
@@ -113,13 +116,36 @@ public static boolean merge(ReduceSinkOperator cRS,
JoinOperator pJoin, int minR
* If parent RS has not been assigned any partitioning column, we will use
* partitioning columns (if exist) of child RS.
*/
- public static boolean merge(ReduceSinkOperator cRS, ReduceSinkOperator pRS,
int minReducer)
+ public static boolean merge(HiveConf hiveConf, ReduceSinkOperator cRS,
ReduceSinkOperator pRS, int minReducer)
throws SemanticException {
int[] result = extractMergeDirections(cRS, pRS, minReducer);
if (result == null) {
return false;
}
+// The partitioning columns of the child RS will replace the columns of the
+// parent RS in two cases:
+// - Parent RS columns are more specific than those of the child RS,
+// and child columns are assigned;
+// - Child RS columns are more specific than those of the parent RS,
+// and parent columns are not assigned.
+List childPCs = cRS.getConf().getPartitionCols();
+List parentPCs = pRS.getConf().getPartitionCols();
+boolean useChildsPartitionColumns =
+result[1] < 0 && (childPCs != null && !childPCs.isEmpty()) ||
+result[1] > 0 && (parentPCs == null || parentPCs.isEmpty());
+
+if (useChildsPartitionColumns) {
+ List newPartitionCols =
ExprNodeDescUtils.backtrack(childPCs, cRS, pRS);
+ long oldParallelism = estimateMaxPartitions(hiveConf, pRS, parentPCs);
+ long newParallelism = estimateMaxPartitions(hiveConf, pRS,
newPartitionCols);
+ long threshold =
hiveConf.getLongVar(HiveConf.ConfVars.HIVEOPTREDUCEDEDUPLICATIONPARALLELISMDECTHRESHOLD);
+ if (oldParallelism / newParallelism > threshold) {
+return false;
+ }
Review comment:
If we don't care about comparing parallelism before/after we could
possibly use the existing `hive.optimize.reducededuplication.min.reducer`
config parameter and not introduce a new one.
```
long newParallelism = estimateMaxPartitions(hiveConf, pRS, newPartitionCols);
if (newParallelism < minReducer) {
return false;
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437802)
Time Spent: 20m (was: 10m)
> Put RS deduplication optimization under cost based decision
> ---
>
> Key: HIVE-23365
> URL: https://issues.apache.org/jira/browse/HIVE-23365
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23365.01.patch, HIVE-23365.02.patch,
> HIVE-23365.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, RS deduplication is always executed whenever it is semantically
> correct. However, it could be beneficial to leave both RS operators in the
> plan, e.g., if the NDV of the second RS is very low. Thus, we would like this
> decision to be cost-based. We could use a simple heuristic that would work
> fine for most of the cases without introducing regressions for existing
> cases, e.g., if NDV for partition column is less than estimated parallelism
> in the second RS, do not execute deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23530) Use SQL functions instead of compute_stats UDAF to compute column statistics
[
https://issues.apache.org/jira/browse/HIVE-23530?focusedWorklogId=437799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437799
]
ASF GitHub Bot logged work on HIVE-23530:
-
Author: ASF GitHub Bot
Created on: 27/May/20 14:10
Start Date: 27/May/20 14:10
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1034:
URL: https://github.com/apache/hive/pull/1034#discussion_r431161927
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
##
@@ -467,6 +467,8 @@
system.registerGenericUDAF("context_ngrams", new
GenericUDAFContextNGrams());
system.registerGenericUDAF("compute_stats", new GenericUDAFComputeStats());
+system.registerGenericUDF("ndv_compute_bit_vector",
GenericUDFNDVComputeBitVector.class);
+system.registerGenericUDAF("compute_bit_vector", new
GenericUDAFComputeBitVector());
Review comment:
I got feedback from Gopal about these names (I used something similar to
your suggestions first) and his reasoning to use these is that they should not
clash with any function that a user has previously added (same for the comment
you left above). That is why they have names that are understandable by us but
not necessarily straightforward. In any case, these are not user-facing
functions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437799)
Time Spent: 0.5h (was: 20m)
> Use SQL functions instead of compute_stats UDAF to compute column statistics
>
>
> Key: HIVE-23530
> URL: https://issues.apache.org/jira/browse/HIVE-23530
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23530.01.patch, HIVE-23530.02.patch,
> HIVE-23530.03.patch, HIVE-23530.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently we compute column statistics by relying on the {{compute_stats}}
> UDAF. For instance, for a given table {{tbl}}, the query to compute
> statistics for columns is translated internally into:
> {code}
> SELECT compute_stats(c1),
>compute_stats(c2),
>...
> FROM tbl;
> {code}
> {{compute_stats}} produces data for the stats available for each column type,
> e.g., struct<"max":long,"min":long,"countnulls":long,...>.
> This issue is to produce a query that relies purely on SQL functions instead:
> {code}
> SELECT max(c1), min(c1), count(case when c1 is null then 1 else null end),
>...
> FROM tbl;
> {code}
> This will allow us to deprecate the {{compute_stats}} UDAF since it mostly
> duplicates functionality found in those other functions. Additionally, many
> of those functions already provide a vectorized implementation so the
> approach can potentially improve the performance of column stats collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23466) ZK registry base should remove only specific instance instead of host
[ https://issues.apache.org/jira/browse/HIVE-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117794#comment-17117794 ] Attila Magyar commented on HIVE-23466: -- The compilation error is because the ExtendedNodeId is only in the latest (0.10.1-SNAPSHOT) TEZ but hive depends on 0.9.1. [~ashutoshc] should we update tez.version in the pom.xml to 0.10.1-SNAPSHOT? > ZK registry base should remove only specific instance instead of host > - > > Key: HIVE-23466 > URL: https://issues.apache.org/jira/browse/HIVE-23466 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23466.1.patch, HIVE-23466.2.patch > > > When ZKRegistryBase detects new ZK nodes it maintains path based cache and > host based cache. The host based cached already handles multiple instances > running in same host. But even if single instance is removed all instances > belonging to the host are removed. > Another issue is that, if single host has multiple instances it returns a Set > with no ordering. Ideally, we want the newest instance to be top of the set > (use TreeSet maybe?). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23466) ZK registry base should remove only specific instance instead of host
[
https://issues.apache.org/jira/browse/HIVE-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117784#comment-17117784
]
Hive QA commented on HIVE-23466:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004120/HIVE-23466.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22646/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22646/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22646/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-27 13:52:22.672
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22646/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-27 13:52:22.675
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f49d257 HIVE-23547 Enforce testconfiguration.properties file
format and alphabetical order (Miklos Gergely, reviewed by Laszlo Bodor)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f49d257 HIVE-23547 Enforce testconfiguration.properties file
format and alphabetical order (Miklos Gergely, reviewed by Laszlo Bodor)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-27 13:52:23.657
+ rm -rf ../yetus_PreCommit-HIVE-Build-22646
+ mkdir ../yetus_PreCommit-HIVE-Build-22646
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22646
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22646/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error:
a/llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java:
does not exist in index
error:
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/ContainerFactory.java:
does not exist in index
error:
a/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java:
does not exist in index
Trying to apply the patch with -p1
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8093298736637076977.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc8093298736637076977.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc5485438094455822296.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
[jira] [Updated] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23462: Status: Patch Available (was: Open) > Add option to rewrite NTILE to sketch functions > --- > > Key: HIVE-23462 > URL: https://issues.apache.org/jira/browse/HIVE-23462 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23462: Attachment: HIVE-23462.02.patch > Add option to rewrite NTILE to sketch functions > --- > > Key: HIVE-23462 > URL: https://issues.apache.org/jira/browse/HIVE-23462 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23462.01.patch, HIVE-23462.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[
https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117781#comment-17117781
]
Hive QA commented on HIVE-23488:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13004116/HIVE-23488.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17286 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22645/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22645/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22645/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13004116 - PreCommit-HIVE-Build
> Optimise PartitionManagementTask::Msck::repair
> --
>
> Key: HIVE-23488
> URL: https://issues.apache.org/jira/browse/HIVE-23488
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23488.1.patch, HIVE-23488.2.patch,
> HIVE-23488.3.patch, Screenshot 2020-05-18 at 5.06.15 AM.png
>
>
> Ends up fetching table information twice.
> !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754!
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java#L113]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java#L234]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23488) Optimise PartitionManagementTask::Msck::repair
[
https://issues.apache.org/jira/browse/HIVE-23488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117764#comment-17117764
]
Hive QA commented on HIVE-23488:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} metastore-server in master failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 1524 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 49 unchanged - 1 fixed = 50 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} metastore-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22645/dev-support/hive-personality.sh
|
| git revision | master / f49d257 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22645/yetus/branch-findbugs-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22645/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22645/yetus/patch-findbugs-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22645/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise PartitionManagementTask::Msck::repair
> --
>
> Key: HIVE-23488
> URL: https://issues.apache.org/jira/browse/HIVE-23488
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23488.1.patch, HIVE-23488.2.patch,
> HIVE-23488.3.patch, Screenshot 2020-05-18 at 5.06.15 AM.png
>
>
> Ends up fetching table information twice.
> !Screenshot 2020-05-18 at 5.06.15 AM.png|width=1084,height=754!
>
>
[jira] [Commented] (HIVE-23363) Upgrade DataNucleus dependency to 5.2
[ https://issues.apache.org/jira/browse/HIVE-23363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117756#comment-17117756 ] David Mollitor commented on HIVE-23363: --- Need the plugin upgraded too please: https://github.com/apache/hive/blob/f76df736d5461c0b6466432522498ca85bd4b240/pom.xml#L1148 https://github.com/apache/hive/blob/270ca800353458ebce6eb262781bd39b15f5e349/standalone-metastore/metastore-server/pom.xml#L712 > Upgrade DataNucleus dependency to 5.2 > - > > Key: HIVE-23363 > URL: https://issues.apache.org/jira/browse/HIVE-23363 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23363.2.patch, HIVE-23363.patch > > > Upgrade Datanucleus from 4.2 to 5.2 as based on it's docs 4.2 has been > retired: > [http://www.datanucleus.org/documentation/products.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-17879) Upgrade Datanucleus Maven Plugin
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117754#comment-17117754
]
David Mollitor commented on HIVE-17879:
---
Must be upgraded to 5.2.0 or higher for "Java 9/10 compatibility to avoid using
add-modules"
http://www.datanucleus.org/documentation/news/access_platform_5_2.html
> Upgrade Datanucleus Maven Plugin
>
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
>
[jira] [Updated] (HIVE-17879) Upgrade Datanucleus Maven Plugin
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-17879:
--
Summary: Upgrade Datanucleus Maven Plugin (was: Can not find java.sql.date
when building hive)
> Upgrade Datanucleus Maven Plugin
>
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:155)
> at
[jira] [Updated] (HIVE-17879) Upgrade Datanucleus Maven Plugin
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-17879:
--
Target Version/s: 4.0.0
> Upgrade Datanucleus Maven Plugin
>
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:155)
> at org.datanucleus.plugin.PluginManager.(PluginManager.java:63)
> at
>
[jira] [Updated] (HIVE-17879) Can not find java.sql.date when building hive
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-17879:
--
Summary: Can not find java.sql.date when building hive (was: Can not find
java.sql.date in JDK9 when building hive)
> Can not find java.sql.date when building hive
> -
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:155)
[jira] [Updated] (HIVE-17879) Can not find java.sql.date in JDK9 when building hive
[
https://issues.apache.org/jira/browse/HIVE-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-17879:
--
Parent Issue: HIVE-22415 (was: HIVE-17632)
> Can not find java.sql.date in JDK9 when building hive
> -
>
> Key: HIVE-17879
> URL: https://issues.apache.org/jira/browse/HIVE-17879
> Project: Hive
> Issue Type: Sub-task
>Reporter: liyunzhang
>Priority: Major
> Attachments: HIVE-17879.patch
>
>
> when build hive with jdk9
> got following error
> {code}
> [ERROR] Failed to execute goal
> org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default) on
> project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer: InvocationTargetException:
> java/sql/Date: java.sql.Date -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.datanucleus:datanucleus-maven-plugin:3.3.0-release:enhance (default)
> on project hive-standalone-metastore: Error executing DataNucleus tool
> org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Error executing
> DataNucleus tool org.datanucleus.enhancer.DataNucleusEnhancer
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:350)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.enhance(AbstractEnhancerMojo.java:266)
> at
> org.datanucleus.maven.AbstractEnhancerMojo.executeDataNucleusTool(AbstractEnhancerMojo.java:72)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.execute(AbstractDataNucleusMojo.java:126)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 more
> Caused by: java.lang.reflect.InvocationTargetException
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.datanucleus.maven.AbstractDataNucleusMojo.executeInJvm(AbstractDataNucleusMojo.java:333)
> ... 25 more
> Caused by: java.lang.NoClassDefFoundError: java/sql/Date
> at org.datanucleus.ClassConstants.(ClassConstants.java:66)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:206)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:155)
> at
[jira] [Updated] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[ https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23555: -- Status: Patch Available (was: Open) > Cancel compaction jobs when hive.compactor.worker.timeout is reached > > > Key: HIVE-23555 > URL: https://issues.apache.org/jira/browse/HIVE-23555 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23555.patch > > > Currently when a compactor worker thread is stuck, or working too long on a > compaction the the initiator might decide to start a new compaction because > of a timeout, but old worker might still wait for the results of the job. > It would be good to cancel the worker as well after the timeout is reached -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23555) Cancel compaction jobs when hive.compactor.worker.timeout is reached
[ https://issues.apache.org/jira/browse/HIVE-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23555: -- Attachment: HIVE-23555.patch > Cancel compaction jobs when hive.compactor.worker.timeout is reached > > > Key: HIVE-23555 > URL: https://issues.apache.org/jira/browse/HIVE-23555 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23555.patch > > > Currently when a compactor worker thread is stuck, or working too long on a > compaction the the initiator might decide to start a new compaction because > of a timeout, but old worker might still wait for the results of the job. > It would be good to cancel the worker as well after the timeout is reached -- This message was sent by Atlassian Jira (v8.3.4#803005)
