[jira] [Commented] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633608#comment-16633608 ] Sankar Hariappan commented on HIVE-20644: - +1, pending tests > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Attachment: HIVE-20644.02 Status: Patch Available (was: Open) [~sankarh], Here's patch with your comment addressed. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01, HIVE-20644.02 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20644) Avoid exposing sensitive infomation through a Hive Runtime exception
[ https://issues.apache.org/jira/browse/HIVE-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20644: -- Status: Open (was: Patch Available) Cancelling .01 patch to attach next version of patch. > Avoid exposing sensitive infomation through a Hive Runtime exception > > > Key: HIVE-20644 > URL: https://issues.apache.org/jira/browse/HIVE-20644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Minor > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-20644.01 > > > The HiveException raised from the following methods is exposing the datarow > the caused the run time exception. > # ReduceRecordSource::GroupIterator::next() - around line 372 > # MapOperator::process() - around line 567 > # ExecReducer::reduce() - around line 243 > In all the cases, a string representation of the row is constructed on the > fly and is included in > the error message. > VectorMapOperator::process() - around line 973 raises the same exception but > it's not exposing the row since the row contents are not included in the > error message. > While trying to reproduce above error, I also found that the arguments to a > UDF get exposed in log messages from FunctionRegistry::invoke() around line > 1114. This too can cause sensitive information to be leaked through error > message. > This way some sensitive information is leaked to a user through exception > message. That information may not be available to the user otherwise. Hence > it's a kind of security breach or violation of access control. > The contents of the row or the arguments to a function may be useful for > debugging and hence it's worth to add those to logs. Hence proposal here to > log a separate message with log level DEBUG or INFO containing the string > representation of the row. Users can configure their logging so that > DEBUG/INFO messages do not go to the client but at the same time are > available in the hive server logs for debugging. The actual exception message > will not contain any sensitive data like row data or argument data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20531: --- Status: Patch Available (was: Open) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch, HIVE-20531.07.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20531: --- Attachment: HIVE-20531.07.patch > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch, HIVE-20531.07.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20531: --- Status: Open (was: Patch Available) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch, HIVE-20531.07.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20623: --- Attachment: HIVE-20623.03.patch > Shared work: Extend sharing of map-join cache entries in LLAP > - > > Key: HIVE-20623 > URL: https://issues.apache.org/jira/browse/HIVE-20623 > Project: Hive > Issue Type: Improvement > Components: llap, Logical Optimizer >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20623.01.patch, HIVE-20623.02.patch, > HIVE-20623.02.patch, HIVE-20623.02.patch, HIVE-20623.03.patch, > HIVE-20623.03.patch, HIVE-20623.03.patch, HIVE-20623.patch, > hash-shared-work.json.txt, hash-shared-work.svg > > > For a query like this > {code} > with all_sales as ( > select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt > as ext_price from store_sales > UNION ALL > select ws_bill_customer_sk as customer_sk, > ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales > UNION ALL > select cs_bill_customer_sk as customer_sk, cs_ext_sales_price - > cs_ext_discount_amt as ext_price from catalog_sales) > select sum(ext_price) total_price, c_customer_id from all_sales, customer > where customer_sk = c_customer_sk > group by c_customer_id > order by total_price desc > limit 100; > {code} > The hashtable used for all 3 joins are identical, which is loaded 3x times in > the same LLAP instance because they are named. > {code} > cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container"; > {code} > in the cache. > If those are identical in nature (i.e vectorization, hashtable type etc), > then the duplication is just wasted CPU, memory and network - using the cache > name for hashtables which will be identical in layout would be extremely > useful. > In cases where the join is pushed through a UNION, those are identical. > This optimization can only be done without concern for accidental delays when > the same upstream task is generating all of these hashtables, which is what > is achieved by the shared scan optimizer already. > In case the shared work is not present, this has potential downsides - in > case two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1 > builder will block the other task from reading from Map 2, even though Map 2 > might have started after, but finished ahead of Map 1. > So this specific optimization can always be considered for cases where the > shared work unifies the operator tree and the parents of all the RS entries > involved are same (& the RS layout is the same). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20636) Improve number of null values estimation after outer join
[ https://issues.apache.org/jira/browse/HIVE-20636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20636: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for reviewing [~ashutoshc] > Improve number of null values estimation after outer join > - > > Key: HIVE-20636 > URL: https://issues.apache.org/jira/browse/HIVE-20636 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20636.01.patch, HIVE-20636.01.patch, > HIVE-20636.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20636) Improve number of null values estimation after outer join
[ https://issues.apache.org/jira/browse/HIVE-20636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633561#comment-16633561 ] Ashutosh Chauhan commented on HIVE-20636: - +1 > Improve number of null values estimation after outer join > - > > Key: HIVE-20636 > URL: https://issues.apache.org/jira/browse/HIVE-20636 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20636.01.patch, HIVE-20636.01.patch, > HIVE-20636.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-20637: Resolution: Fixed Fix Version/s: (was: 3.0.1) 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Miklos! > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20637.01.patch, HIVE-20637.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633541#comment-16633541 ] Ashutosh Chauhan commented on HIVE-20637: - +1 > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20637.01.patch, HIVE-20637.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[ https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20660: --- Attachment: HIVE-20660.1.patch > Group by statistics estimation could be improved by bounding the total number > of rows to source table > - > > Key: HIVE-20660 > URL: https://issues.apache.org/jira/browse/HIVE-20660 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20660.1.patch > > > Currently the stats for group by is estimated by taking product of NDVs of > all the keys and bounding it by the number of rows of its input. This bound > could be improved by using the source table instead of immediate input, the > insight in this case is that cardinality/ndvs of a table can not go beyond > the original (outer joins will only add NULLs thereby increasing the > cardinality by 1). > Note that the assumption here is that group by keys all belong to the same > source table/input. > This will improve the estimation in situations where group by is executed > after joins wherein Hive could end up estimating the number of rows. > *Reproducer* > {code:sql} > set hive.stats.fetch.column.stats=true; > create table t1(i int, j int); > alter table t1 update statistics set('numRows'='1', > 'rawDataSize'='18000'); > alter table t1 update statistics for column i > set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0'); > alter table t1 update statistics for column j > set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50'); > create table t2(i2 int, j2 int); > alter table t2 update statistics set('numRows'='1', > 'rawDataSize'='1'); > alter table t2 update statistics for column i2 > set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0'); > alter table t2 update statistics for column j2 > set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1'); > explain select count (1) from t1,t2 > where t1.j=t2.i2 > group by t1.i, t1.j; > {code} > {code:sql} > Reducer 2 > Reduce Operator Tree: > Merge Join Operator > condition map: > Inner Join 0 to 1 > keys: > 0 _col1 (type: int) > 1 _col0 (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 99700 Data size: 797288 Basic stats: > COMPLETE Column stats: COMPLETE > Group By Operator > aggregations: count() > keys: _col0 (type: int), _col1 (type: int) > mode: hash > outputColumnNames: _col0, _col1, _col2 > Statistics: Num rows: 49850 Data size: 797448 Basic stats: > COMPLETE Column stats: COMPLETE <== > Reduce Output Operator > key expressions: _col0 (type: int), _col1 (type: int) > sort order: ++ > Map-reduce partition columns: _col0 (type: int), _col1 > (type: int) > Statistics: Num rows: 49850 Data size: 797448 Basic > stats: COMPLETE Column stats: COMPLETE > value expressions: _col2 (type: bigint) > . > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[ https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20660: --- Status: Patch Available (was: Open) > Group by statistics estimation could be improved by bounding the total number > of rows to source table > - > > Key: HIVE-20660 > URL: https://issues.apache.org/jira/browse/HIVE-20660 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20660.1.patch > > > Currently the stats for group by is estimated by taking product of NDVs of > all the keys and bounding it by the number of rows of its input. This bound > could be improved by using the source table instead of immediate input, the > insight in this case is that cardinality/ndvs of a table can not go beyond > the original (outer joins will only add NULLs thereby increasing the > cardinality by 1). > Note that the assumption here is that group by keys all belong to the same > source table/input. > This will improve the estimation in situations where group by is executed > after joins wherein Hive could end up estimating the number of rows. > *Reproducer* > {code:sql} > set hive.stats.fetch.column.stats=true; > create table t1(i int, j int); > alter table t1 update statistics set('numRows'='1', > 'rawDataSize'='18000'); > alter table t1 update statistics for column i > set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0'); > alter table t1 update statistics for column j > set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50'); > create table t2(i2 int, j2 int); > alter table t2 update statistics set('numRows'='1', > 'rawDataSize'='1'); > alter table t2 update statistics for column i2 > set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0'); > alter table t2 update statistics for column j2 > set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1'); > explain select count (1) from t1,t2 > where t1.j=t2.i2 > group by t1.i, t1.j; > {code} > {code:sql} > Reducer 2 > Reduce Operator Tree: > Merge Join Operator > condition map: > Inner Join 0 to 1 > keys: > 0 _col1 (type: int) > 1 _col0 (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 99700 Data size: 797288 Basic stats: > COMPLETE Column stats: COMPLETE > Group By Operator > aggregations: count() > keys: _col0 (type: int), _col1 (type: int) > mode: hash > outputColumnNames: _col0, _col1, _col2 > Statistics: Num rows: 49850 Data size: 797448 Basic stats: > COMPLETE Column stats: COMPLETE <== > Reduce Output Operator > key expressions: _col0 (type: int), _col1 (type: int) > sort order: ++ > Map-reduce partition columns: _col0 (type: int), _col1 > (type: int) > Statistics: Num rows: 49850 Data size: 797448 Basic > stats: COMPLETE Column stats: COMPLETE > value expressions: _col2 (type: bigint) > . > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20660) Group by statistics estimation could be improved by bounding the total number of rows to source table
[ https://issues.apache.org/jira/browse/HIVE-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20660: -- > Group by statistics estimation could be improved by bounding the total number > of rows to source table > - > > Key: HIVE-20660 > URL: https://issues.apache.org/jira/browse/HIVE-20660 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > Currently the stats for group by is estimated by taking product of NDVs of > all the keys and bounding it by the number of rows of its input. This bound > could be improved by using the source table instead of immediate input, the > insight in this case is that cardinality/ndvs of a table can not go beyond > the original (outer joins will only add NULLs thereby increasing the > cardinality by 1). > Note that the assumption here is that group by keys all belong to the same > source table/input. > This will improve the estimation in situations where group by is executed > after joins wherein Hive could end up estimating the number of rows. > *Reproducer* > {code:sql} > set hive.stats.fetch.column.stats=true; > create table t1(i int, j int); > alter table t1 update statistics set('numRows'='1', > 'rawDataSize'='18000'); > alter table t1 update statistics for column i > set('numDVs'='2500','numNulls'='50','highValue'='1000','lowValue'='0'); > alter table t1 update statistics for column j > set('numDVs'='500','numNulls'='30','highValue'='100','lowValue'='50'); > create table t2(i2 int, j2 int); > alter table t2 update statistics set('numRows'='1', > 'rawDataSize'='1'); > alter table t2 update statistics for column i2 > set('numDVs'='1000','numNulls'='0','highValue'='8000','lowValue'='0'); > alter table t2 update statistics for column j2 > set('numDVs'='10','numNulls'='0','highValue'='800','lowValue'='-1'); > explain select count (1) from t1,t2 > where t1.j=t2.i2 > group by t1.i, t1.j; > {code} > {code:sql} > Reducer 2 > Reduce Operator Tree: > Merge Join Operator > condition map: > Inner Join 0 to 1 > keys: > 0 _col1 (type: int) > 1 _col0 (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 99700 Data size: 797288 Basic stats: > COMPLETE Column stats: COMPLETE > Group By Operator > aggregations: count() > keys: _col0 (type: int), _col1 (type: int) > mode: hash > outputColumnNames: _col0, _col1, _col2 > Statistics: Num rows: 49850 Data size: 797448 Basic stats: > COMPLETE Column stats: COMPLETE <== > Reduce Output Operator > key expressions: _col0 (type: int), _col1 (type: int) > sort order: ++ > Map-reduce partition columns: _col0 (type: int), _col1 > (type: int) > Statistics: Num rows: 49850 Data size: 797448 Basic > stats: COMPLETE Column stats: COMPLETE > value expressions: _col2 (type: bigint) > . > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633515#comment-16633515 ] Hive QA commented on HIVE-20653: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941900/HIVE-20653.4.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15009 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14150/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14150/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14150/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941900 - PreCommit-HIVE-Build > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch, HIVE-20653.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20638: Attachment: HIVE-20638.03.patch > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch, > HIVE-20638.03.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633505#comment-16633505 ] Hive QA commented on HIVE-20653: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 2m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14150/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | modules | C: metastore ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14150/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch, HIVE-20653.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633500#comment-16633500 ] Hive QA commented on HIVE-20638: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941896/HIVE-20638.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15009 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14149/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14149/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14149/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941896 - PreCommit-HIVE-Build > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633495#comment-16633495 ] Hive QA commented on HIVE-20638: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 20s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 27s{color} | {color:red} root in master failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 26s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 26s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 8s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 39m 34s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc xml compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14149/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14149/yetus/branch-compile-root.txt | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14149/yetus/patch-compile-root.txt | | javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-14149/yetus/patch-compile-root.txt | | modules | C: serde . hbase-handler hcatalog/webhcat/svr U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14149/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20544) TOpenSessionReq logs password and username
[ https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633484#comment-16633484 ] Hive QA commented on HIVE-20544: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941893/HIVE-20544.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15010 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=252) org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover (batchId=252) org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery (batchId=252) org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=252) org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized (batchId=252) org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=252) org.apache.hive.jdbc.miniHS2.TestHs2ConnectionMetricsBinary.testOpenConnectionMetrics (batchId=256) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14148/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14148/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14148/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941893 - PreCommit-HIVE-Build > TOpenSessionReq logs password and username > -- > > Key: HIVE-20544 > URL: https://issues.apache.org/jira/browse/HIVE-20544 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, patch, security > Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch, > HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch, > HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.patch, > non-solution.patch, working-solution.patch > > > In > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq, > if client protocol is unset, validate() and toString() prints both username > and password to logs. > Logging a password is a security risk. We should hide the ***. > =Edit= (no longer relevant, see comments) > This issue is tricky since it is caused in a fully generated class. I've been > playing around and have found one working solution, butI'd truly appreciate > ideas for a more elegant solution or input. > The problem: > TCLIService.thrift is the template for generating all classes in > service-rpc. Struct TOpenSessionReq is OpenSession()'s one parameter and is > defined thus: > {noformat} > struct TOpenSessionReq { > 1: required TProtocolVersion client_protocol = > TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10 > 2: optional string username > 3: optional string password > 4: optional map configuration > } > {noformat} > In the generated class TOpenSessionReq.java, client_protocol is checked by a > validate() method, which is called quite a few times; if client_protocol is > not set, it throws a TProtocolException, passing along a toString(). This > toString() gets the names and values of all fields, including username and > password. > Working solution: > * Create a separate struct containing only the username and password, and > pass it to OpenSession() as a second parameter. Since all fields in the new > struct are "optional", the generated validate() is empty – toString() is > never used. This involves changing core classes and breaks the "Each function > should take exactly one parameter" coding convention (detailed at > service-rpc/if/TCLIService.thrift:27). > See working-solution.patch. > What doesn't work: > * Making client_protocol optional instead of required. Apparently this will > break everything. > * Overwriting toString() – TOpenSessionReq is a struct. > * Creating two Thrift structs, one struct for required (TRequiredReq) and > one for optional (TOptionalReq) fields, and nesting them in struct > TOpenSessionReq. This doesn't work because validate() in TOpenSessionReq can > call TOptionalReq.toString(), which prints the password to logs. This will > happen if TRequiredReq.client_protocol isn't set. > See
[jira] [Commented] (HIVE-20544) TOpenSessionReq logs password and username
[ https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633469#comment-16633469 ] Hive QA commented on HIVE-20544: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 59s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile xml | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14148/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14148/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: service-rpc itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14148/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > TOpenSessionReq logs password and username > -- > > Key: HIVE-20544 > URL: https://issues.apache.org/jira/browse/HIVE-20544 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, patch, security > Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch, > HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch, > HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.patch, > non-solution.patch, working-solution.patch > > > In > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq, > if client protocol is unset, validate() and toString() prints both username > and password to logs. > Logging a password is a security risk. We should hide the ***. > =Edit= (no longer relevant, see comments) > This issue is
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633462#comment-16633462 ] Hive QA commented on HIVE-17300: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941892/HIVE-17300.10.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15011 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14147/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14147/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14147/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941892 - PreCommit-HIVE-Build > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Attachments: HIVE-17300.10.patch, HIVE-17300.10.patch, > HIVE-17300.3.patch, HIVE-17300.4.patch, HIVE-17300.5.patch, > HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.7.patch, > HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.8.patch, > HIVE-17300.8.patch, HIVE-17300.9.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20653: -- Attachment: HIVE-20653.4.patch > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch, HIVE-20653.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633451#comment-16633451 ] Hive QA commented on HIVE-17300: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} service in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 34m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14147/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: common itests/hive-unit ql service U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14147/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Attachments: HIVE-17300.10.patch, HIVE-17300.10.patch, > HIVE-17300.3.patch, HIVE-17300.4.patch, HIVE-17300.5.patch, > HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.7.patch, > HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.8.patch, > HIVE-17300.8.patch, HIVE-17300.9.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > >
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633439#comment-16633439 ] Hive QA commented on HIVE-20535: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941891/HIVE-20535.19.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15004 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=194) [druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q] org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14146/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14146/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14146/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941891 - PreCommit-HIVE-Build > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch, > HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch, > HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch, > HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch, > HIVE-20535.2.patch, HIVE-20535.3.patch, HIVE-20535.4.patch, > HIVE-20535.5.patch, HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633419#comment-16633419 ] Hive QA commented on HIVE-20535: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 24s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 55s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 2 new + 142 unchanged - 6 fixed = 144 total (was 148) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14146/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14146/yetus/diff-checkstyle-ql.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14146/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch, > HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch, > HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch, > HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch, > HIVE-20535.2.patch, HIVE-20535.3.patch, HIVE-20535.4.patch, > HIVE-20535.5.patch, HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633408#comment-16633408 ] Hive QA commented on HIVE-20647: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941890/HIVE-20647.4.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15009 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=168) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14145/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14145/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14145/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941890 - PreCommit-HIVE-Build > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633394#comment-16633394 ] Hive QA commented on HIVE-20647: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} itests/util in master has 52 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} itests/util: The patch generated 0 new + 56 unchanged - 4 fixed = 56 total (was 60) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} The patch hive-unit passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} itests/util generated 0 new + 50 unchanged - 2 fixed = 50 total (was 52) {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} hive-unit in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 18m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14145/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: itests/util itests/hive-unit U: itests | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14145/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633386#comment-16633386 ] Hive QA commented on HIVE-20637: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941888/HIVE-20637.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15007 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14144/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14144/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14144/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941888 - PreCommit-HIVE-Build > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20637.01.patch, HIVE-20637.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633381#comment-16633381 ] Hive QA commented on HIVE-20637: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 27s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 26s{color} | {color:red} root in master failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 54s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 31s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 31s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} The patch . passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} ql: The patch generated 0 new + 808 unchanged - 3 fixed = 808 total (was 811) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s{color} | {color:green} ql generated 0 new + 2321 unchanged - 1 fixed = 2321 total (was 2322) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 36s{color} | {color:red} root generated 1 new + 387 unchanged - 1 fixed = 388 total (was 388) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 57s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 50m 52s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14144/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus/branch-compile-root.txt | | findbugs | v3.0.0 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus/patch-compile-root.txt | | javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus/patch-compile-root.txt | | javadoc | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus/diff-javadoc-javadoc-root.txt | | javadoc | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus/diff-javadoc-javadoc-ql.txt | | modules | C: . ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14144/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos
[jira] [Updated] (HIVE-20638) Upgrade version of Jetty to 9.3.25.v20180904
[ https://issues.apache.org/jira/browse/HIVE-20638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20638: Attachment: HIVE-20638.02.patch > Upgrade version of Jetty to 9.3.25.v20180904 > > > Key: HIVE-20638 > URL: https://issues.apache.org/jira/browse/HIVE-20638 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20638.01.patch, HIVE-20638.02.patch > > > Current version is 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633370#comment-16633370 ] Hive QA commented on HIVE-20609: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941889/HIVE-20609.04.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15012 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14143/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14143/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14143/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941889 - PreCommit-HIVE-Build > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch, HIVE-20609.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633359#comment-16633359 ] Hive QA commented on HIVE-20609: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} llap-server in master has 84 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14143/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: llap-server U: llap-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14143/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch, HIVE-20609.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20544) TOpenSessionReq logs password and username
[ https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-20544: - Status: Open (was: Patch Available) > TOpenSessionReq logs password and username > -- > > Key: HIVE-20544 > URL: https://issues.apache.org/jira/browse/HIVE-20544 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, patch, security > Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch, > HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch, > HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.patch, > non-solution.patch, working-solution.patch > > > In > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq, > if client protocol is unset, validate() and toString() prints both username > and password to logs. > Logging a password is a security risk. We should hide the ***. > =Edit= (no longer relevant, see comments) > This issue is tricky since it is caused in a fully generated class. I've been > playing around and have found one working solution, butI'd truly appreciate > ideas for a more elegant solution or input. > The problem: > TCLIService.thrift is the template for generating all classes in > service-rpc. Struct TOpenSessionReq is OpenSession()'s one parameter and is > defined thus: > {noformat} > struct TOpenSessionReq { > 1: required TProtocolVersion client_protocol = > TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10 > 2: optional string username > 3: optional string password > 4: optional map configuration > } > {noformat} > In the generated class TOpenSessionReq.java, client_protocol is checked by a > validate() method, which is called quite a few times; if client_protocol is > not set, it throws a TProtocolException, passing along a toString(). This > toString() gets the names and values of all fields, including username and > password. > Working solution: > * Create a separate struct containing only the username and password, and > pass it to OpenSession() as a second parameter. Since all fields in the new > struct are "optional", the generated validate() is empty – toString() is > never used. This involves changing core classes and breaks the "Each function > should take exactly one parameter" coding convention (detailed at > service-rpc/if/TCLIService.thrift:27). > See working-solution.patch. > What doesn't work: > * Making client_protocol optional instead of required. Apparently this will > break everything. > * Overwriting toString() – TOpenSessionReq is a struct. > * Creating two Thrift structs, one struct for required (TRequiredReq) and > one for optional (TOptionalReq) fields, and nesting them in struct > TOpenSessionReq. This doesn't work because validate() in TOpenSessionReq can > call TOptionalReq.toString(), which prints the password to logs. This will > happen if TRequiredReq.client_protocol isn't set. > See non-solution.patch > * Asking Thrift devs to change their code. I wrote them an email but have no > expectations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20544) TOpenSessionReq logs password and username
[ https://issues.apache.org/jira/browse/HIVE-20544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-20544: - Attachment: HIVE-20544.4.patch Status: Patch Available (was: Open) > TOpenSessionReq logs password and username > -- > > Key: HIVE-20544 > URL: https://issues.apache.org/jira/browse/HIVE-20544 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, patch, security > Attachments: HIVE-20544.1.patch, HIVE-20544.2.patch, > HIVE-20544.3.patch, HIVE-20544.3.patch, HIVE-20544.4.patch, > HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.4.patch, HIVE-20544.patch, > non-solution.patch, working-solution.patch > > > In > service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq, > if client protocol is unset, validate() and toString() prints both username > and password to logs. > Logging a password is a security risk. We should hide the ***. > =Edit= (no longer relevant, see comments) > This issue is tricky since it is caused in a fully generated class. I've been > playing around and have found one working solution, butI'd truly appreciate > ideas for a more elegant solution or input. > The problem: > TCLIService.thrift is the template for generating all classes in > service-rpc. Struct TOpenSessionReq is OpenSession()'s one parameter and is > defined thus: > {noformat} > struct TOpenSessionReq { > 1: required TProtocolVersion client_protocol = > TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10 > 2: optional string username > 3: optional string password > 4: optional map configuration > } > {noformat} > In the generated class TOpenSessionReq.java, client_protocol is checked by a > validate() method, which is called quite a few times; if client_protocol is > not set, it throws a TProtocolException, passing along a toString(). This > toString() gets the names and values of all fields, including username and > password. > Working solution: > * Create a separate struct containing only the username and password, and > pass it to OpenSession() as a second parameter. Since all fields in the new > struct are "optional", the generated validate() is empty – toString() is > never used. This involves changing core classes and breaks the "Each function > should take exactly one parameter" coding convention (detailed at > service-rpc/if/TCLIService.thrift:27). > See working-solution.patch. > What doesn't work: > * Making client_protocol optional instead of required. Apparently this will > break everything. > * Overwriting toString() – TOpenSessionReq is a struct. > * Creating two Thrift structs, one struct for required (TRequiredReq) and > one for optional (TOptionalReq) fields, and nesting them in struct > TOpenSessionReq. This doesn't work because validate() in TOpenSessionReq can > call TOptionalReq.toString(), which prints the password to logs. This will > happen if TRequiredReq.client_protocol isn't set. > See non-solution.patch > * Asking Thrift devs to change their code. I wrote them an email but have no > expectations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Attachment: HIVE-17300.10.patch Status: Patch Available (was: Open) > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Attachments: HIVE-17300.10.patch, HIVE-17300.10.patch, > HIVE-17300.3.patch, HIVE-17300.4.patch, HIVE-17300.5.patch, > HIVE-17300.6.patch, HIVE-17300.7.patch, HIVE-17300.7.patch, > HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.8.patch, > HIVE-17300.8.patch, HIVE-17300.9.patch, HIVE-17300.patch, > complete_success.png, full_mapred_stats.png, graph_with_mapred_stats.png, > last_stage_error.png, last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-17300: - Status: Open (was: Patch Available) > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Sub-task > Components: Web UI >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Labels: beginner, features, patch > Attachments: HIVE-17300.10.patch, HIVE-17300.3.patch, > HIVE-17300.4.patch, HIVE-17300.5.patch, HIVE-17300.6.patch, > HIVE-17300.7.patch, HIVE-17300.7.patch, HIVE-17300.8.patch, > HIVE-17300.8.patch, HIVE-17300.8.patch, HIVE-17300.8.patch, > HIVE-17300.9.patch, HIVE-17300.patch, complete_success.png, > full_mapred_stats.png, graph_with_mapred_stats.png, last_stage_error.png, > last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20535) Add new configuration to set the size of the global compile lock
[ https://issues.apache.org/jira/browse/HIVE-20535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20535: -- Attachment: HIVE-20535.19.patch > Add new configuration to set the size of the global compile lock > > > Key: HIVE-20535 > URL: https://issues.apache.org/jira/browse/HIVE-20535 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20535.1.patch, HIVE-20535.10.patch, > HIVE-20535.11.patch, HIVE-20535.12.patch, HIVE-20535.13.patch, > HIVE-20535.14.patch, HIVE-20535.15.patch, HIVE-20535.16.patch, > HIVE-20535.17.patch, HIVE-20535.18.patch, HIVE-20535.19.patch, > HIVE-20535.2.patch, HIVE-20535.3.patch, HIVE-20535.4.patch, > HIVE-20535.5.patch, HIVE-20535.6.patch, HIVE-20535.8.patch, HIVE-20535.9.patch > > > When removing the compile lock, it is quite risky to remove it entirely. > It would be good to provide a pool size for the concurrent compilation, so > the administrator can limit the load -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20647) HadoopVer was ignored in QTestUtil
[ https://issues.apache.org/jira/browse/HIVE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] denys kuzmenko updated HIVE-20647: -- Attachment: HIVE-20647.4.patch > HadoopVer was ignored in QTestUtil > -- > > Key: HIVE-20647 > URL: https://issues.apache.org/jira/browse/HIVE-20647 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: denys kuzmenko >Assignee: denys kuzmenko >Priority: Major > Attachments: HIVE-20647.1.patch, HIVE-20647.2.patch, > HIVE-20647.3.patch, HIVE-20647.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20609: -- Status: Patch Available (was: Open) re-submitting the same patch > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch, HIVE-20609.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20609: -- Attachment: HIVE-20609.04.patch > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch, HIVE-20609.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20609) Create SSD cache dir if it doesnt exist already
[ https://issues.apache.org/jira/browse/HIVE-20609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20609: -- Status: Open (was: Patch Available) > Create SSD cache dir if it doesnt exist already > --- > > Key: HIVE-20609 > URL: https://issues.apache.org/jira/browse/HIVE-20609 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20609.01.patch, HIVE-20609.02.patch, > HIVE-20609.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20637: -- Status: Patch Available (was: Open) re-submitting the same patch > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20637.01.patch, HIVE-20637.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20637: -- Attachment: HIVE-20637.02.patch > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20637.01.patch, HIVE-20637.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20637) Allow any udfs with 0 arguments or with constant arguments as part of default clause
[ https://issues.apache.org/jira/browse/HIVE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20637: -- Status: Open (was: Patch Available) > Allow any udfs with 0 arguments or with constant arguments as part of default > clause > > > Key: HIVE-20637 > URL: https://issues.apache.org/jira/browse/HIVE-20637 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 3.0.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 3.0.1 > > Attachments: HIVE-20637.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633353#comment-16633353 ] Hive QA commented on HIVE-20653: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941871/HIVE-20653.3.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15009 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=168) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14142/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14142/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14142/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941871 - PreCommit-HIVE-Build > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633346#comment-16633346 ] Hive QA commented on HIVE-20653: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 2m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14142/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | modules | C: metastore ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14142/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[ https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633343#comment-16633343 ] Hive QA commented on HIVE-20651: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941870/HIVE-20651.3.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15010 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14141/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14141/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14141/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941870 - PreCommit-HIVE-Build > JdbcStorageHandler password should be encrypted > --- > > Key: HIVE-20651 > URL: https://issues.apache.org/jira/browse/HIVE-20651 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch, > HIVE-20651.3.patch > > > Currently, external jdbc table with JdbcStorageHandler store password as > "hive.sql.dbcp.password" table property in clear text. We should put it in a > keystore file. Here is the proposed change: > {code:java} > …. > STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' > TBLPROPERTIES ( > "hive.sql.dbcp.password.keystore" = > "hdfs:///user/hive/credential/postgres.jceks", > "hive.sql.dbcp.password.key" = "mydb.password" > ); > {code} > > The jceks file is created with: > {code} > hadoop credential create mydb.password -provider > hdfs:///user/hive/credential/postgres.jceks -v secretpassword > {code} > User can choose to put all db password in one jceks, or a separate jceks for > each db. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20651) JdbcStorageHandler password should be encrypted
[ https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633342#comment-16633342 ] Hive QA commented on HIVE-20651: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 36s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 23s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 30s{color} | {color:red} root in master failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 50s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 20s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 37s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 37s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} jdbc-handler: The patch generated 5 new + 27 unchanged - 2 fixed = 32 total (was 29) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 3 new + 374 unchanged - 0 fixed = 377 total (was 374) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s{color} | {color:green} jdbc-handler generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 4s{color} | {color:red} ql generated 2 new + 2320 unchanged - 2 fixed = 2322 total (was 2322) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Found reliance on default encoding in org.apache.hadoop.hive.ql.exec.Utilities.copyJobSecretToTableProperties(TableDesc):in org.apache.hadoop.hive.ql.exec.Utilities.copyJobSecretToTableProperties(TableDesc): new String(byte[]) At Utilities.java:[line 2315] | | | Found reliance on default encoding in org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobConf(TableDesc, JobConf):in org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobConf(TableDesc, JobConf): String.getBytes() At PlanUtils.java:[line 989] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc xml compile findbugs checkstyle | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14141/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14141/yetus/branch-compile-root.txt | | findbugs | v3.0.0 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14141/yetus/patch-compile-root.txt | | javac |
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633324#comment-16633324 ] Hive QA commented on HIVE-20531: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941865/HIVE-20531.06.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15010 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14140/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14140/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14140/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941865 - PreCommit-HIVE-Build > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633315#comment-16633315 ] Hive QA commented on HIVE-20531: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 33s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 57s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 680 unchanged - 0 fixed = 683 total (was 680) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 1 new + 144 unchanged - 1 fixed = 145 total (was 145) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 42s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14140/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14140/yetus/diff-checkstyle-itests_hive-unit.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14140/yetus/diff-checkstyle-ql.txt | | modules | C: common itests/hive-unit ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14140/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > >
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633301#comment-16633301 ] Hive QA commented on HIVE-20646: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941864/HIVE-20646.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15009 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14139/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14139/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14139/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941864 - PreCommit-HIVE-Build > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20653: -- Attachment: HIVE-20653.3.patch > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20651) JdbcStorageHandler password should be encrypted
[ https://issues.apache.org/jira/browse/HIVE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20651: -- Attachment: HIVE-20651.3.patch > JdbcStorageHandler password should be encrypted > --- > > Key: HIVE-20651 > URL: https://issues.apache.org/jira/browse/HIVE-20651 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20651.1.patch, HIVE-20651.2.patch, > HIVE-20651.3.patch > > > Currently, external jdbc table with JdbcStorageHandler store password as > "hive.sql.dbcp.password" table property in clear text. We should put it in a > keystore file. Here is the proposed change: > {code:java} > …. > STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' > TBLPROPERTIES ( > "hive.sql.dbcp.password.keystore" = > "hdfs:///user/hive/credential/postgres.jceks", > "hive.sql.dbcp.password.key" = "mydb.password" > ); > {code} > > The jceks file is created with: > {code} > hadoop credential create mydb.password -provider > hdfs:///user/hive/credential/postgres.jceks -v secretpassword > {code} > User can choose to put all db password in one jceks, or a separate jceks for > each db. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633298#comment-16633298 ] Hive QA commented on HIVE-20646: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 23s{color} | {color:blue} standalone-metastore/metastore-common in master has 28 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 49s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 5s{color} | {color:blue} standalone-metastore/metastore-server in master has 182 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} The patch metastore-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} ql: The patch generated 0 new + 52 unchanged - 1 fixed = 52 total (was 53) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} The patch metastore-server passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 20s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14139/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-common ql standalone-metastore/metastore-server U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14139/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore,
[jira] [Assigned] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan reassigned HIVE-20531: --- Assignee: mahesh kumar behera (was: Sankar Hariappan) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20531: Status: Patch Available (was: Open) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20531: Attachment: HIVE-20531.06.patch > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Attachment: HIVE-20646.02.patch > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20531: Attachment: (was: HIVE-20531.06.patch) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Patch Available (was: Open) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch, HIVE-20646.02.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20531) Repl load on cloud storage file system can skip redundant move or add partition tasks.
[ https://issues.apache.org/jira/browse/HIVE-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20531: Assignee: Sankar Hariappan (was: mahesh kumar behera) Status: Open (was: Patch Available) > Repl load on cloud storage file system can skip redundant move or add > partition tasks. > -- > > Key: HIVE-20531 > URL: https://issues.apache.org/jira/browse/HIVE-20531 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20531.01.patch, HIVE-20531.02.patch, > HIVE-20531.03.patch, HIVE-20531.04.patch, HIVE-20531.05.patch, > HIVE-20531.06.patch > > > In replication load, both add partition and insert operations are handled > through import. Import creates 3 major tasks. Copy, add partition and move. > Copy does the copy of data from source location to staging directory. Then > add partition (which runs in parallel to copy) creates the partition in meta > store. Its a no op in case of insert and by the time this ddl task is > executed for insert partition would be already present. The third operation > is move. Which actually moves the file from staging directory to actual > location. And then in case of insert it adds the insert event to notification > table. It does this for add partition operation which is redundant as the > event for add partition would have been written already by ddl task. With the > optimization to copy directly to actual table location in S3, move task can > be avoided for add partition operation replay and replay of insert need not > create the add partition (ddl) task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Attachment: (was: HIVE-20646.02.patch) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.
[ https://issues.apache.org/jira/browse/HIVE-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-20646: Status: Open (was: Patch Available) > Partition filter condition is not pushed down to metastore query if it has IS > NOT NULL. > --- > > Key: HIVE-20646 > URL: https://issues.apache.org/jira/browse/HIVE-20646 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Metastore, Performance, pull-request-available > Attachments: HIVE-20646.01.patch > > > If the partition filter condition has "is not null" then the filter query > isn't getting pushed to the SQL query in RDMBS. > This slows down metastore api calls for getting list of partitions with > filter condition. > This condition gets added by optimizer in many cases so this is affecting > many queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633269#comment-16633269 ] Hive QA commented on HIVE-20556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 35s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 40s{color} | {color:red} root in master failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 36s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 33s{color} | {color:blue} standalone-metastore/metastore-common in master has 28 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 5s{color} | {color:blue} ql in master has 2322 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 8s{color} | {color:blue} standalone-metastore/metastore-server in master has 182 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 41s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 41s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14138/dev-support/hive-personality.sh | | git revision | master / e133ec5 | | Default Java | 1.8.0_111 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14138/yetus/branch-compile-root.txt | | findbugs | v3.0.0 | | compile | http://104.198.109.242/logs//PreCommit-HIVE-Build-14138/yetus/patch-compile-root.txt | | javac | http://104.198.109.242/logs//PreCommit-HIVE-Build-14138/yetus/patch-compile-root.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-14138/yetus/whitespace-eol.txt | | modules | C: standalone-metastore/metastore-common . itests/hive-unit ql standalone-metastore/metastore-server U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14138/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive >
[jira] [Commented] (HIVE-20556) Expose an API to retrieve the TBL_ID from TBLS in the metastore tables
[ https://issues.apache.org/jira/browse/HIVE-20556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633264#comment-16633264 ] Hive QA commented on HIVE-20556: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12941860/HIVE-20556.17.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15013 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part3] (batchId=160) org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands (batchId=204) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14138/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14138/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14138/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12941860 - PreCommit-HIVE-Build > Expose an API to retrieve the TBL_ID from TBLS in the metastore tables > -- > > Key: HIVE-20556 > URL: https://issues.apache.org/jira/browse/HIVE-20556 > Project: Hive > Issue Type: New Feature > Components: Metastore, Standalone Metastore >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20556.1.patch, HIVE-20556.10.patch, > HIVE-20556.11.patch, HIVE-20556.12.patch, HIVE-20556.13.patch, > HIVE-20556.14.patch, HIVE-20556.15.patch, HIVE-20556.16.patch, > HIVE-20556.17.patch, HIVE-20556.2.patch, HIVE-20556.3.patch, > HIVE-20556.4.patch, HIVE-20556.5.patch, HIVE-20556.6.patch, > HIVE-20556.7.patch, HIVE-20556.8.patch, HIVE-20556.9.patch > > > We have two options to do this > 1) Use the current MTable and add a field for this value > 2) Add an independent API call to the metastore that would return the TBL_ID. > Option 1 is preferable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)