[jira] [Work logged] (HIVE-25875) Support multiple authentication mechanisms simultaneously
[
https://issues.apache.org/jira/browse/HIVE-25875?focusedWorklogId=726145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726145
]
ASF GitHub Bot logged work on HIVE-25875:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 07:06
Start Date: 14/Feb/22 07:06
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera closed pull request #2954:
URL: https://github.com/apache/hive/pull/2954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726145)
Time Spent: 1h 10m (was: 1h)
> Support multiple authentication mechanisms simultaneously
> --
>
> Key: HIVE-25875
> URL: https://issues.apache.org/jira/browse/HIVE-25875
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently, HS2 supports a single form of auth on any given instance of
> HiveServer2. Hive should be able to support multiple auth mechanisms on a
> single instance especially with http transport. for example, LDAP and SAML.
> In both cases, HS2 ends up with receiving an Authorization header in the
> request.
> Similarly, we could be able to support JWT support or other forms of boundary
> authentication that is done outside of Hive (This is not in the scope of this
> jira).
> *The current scope of the Jira:* Support SAML and LDAP authentication modes
> simultaneously when the transport mode is set to *HTTP.* This multi-auth will
> not work when transport mode is set to {_}binary or all{_}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-25912.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master.
Thanks for the patch [~fcbai]!
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2, 4.0.0
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Fix For: 4.0.0
>
> Attachments: hive bugs.png, hive-bug-01.png
>
> Original Estimate: 96h
> Time Spent: 17h
> Remaining Estimate: 79h
>
> *new update:*
> I test the master branch, have the same problem.
> --
> ENV:
> Hive 3.1.2
> HDFS:3.3.1
> enable OpenLDAP and Ranger .
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?focusedWorklogId=726144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726144
]
ASF GitHub Bot logged work on HIVE-25912:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 06:49
Start Date: 14/Feb/22 06:49
Worklog Time Spent: 10m
Work Description: pvary merged pull request #3009:
URL: https://github.com/apache/hive/pull/3009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726144)
Remaining Estimate: 79h (was: 79h 10m)
Time Spent: 17h (was: 16h 50m)
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2, 4.0.0
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Attachments: hive bugs.png, hive-bug-01.png
>
> Original Estimate: 96h
> Time Spent: 17h
> Remaining Estimate: 79h
>
> *new update:*
> I test the master branch, have the same problem.
> --
> ENV:
> Hive 3.1.2
> HDFS:3.3.1
> enable OpenLDAP and Ranger .
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-25895. - Fix Version/s: 4.0.0 Resolution: Fixed > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491810#comment-17491810 ] Ayush Saxena commented on HIVE-25895: - Merged PR to master. Thanx [~pkumarsinha] for the review!!! > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?focusedWorklogId=726140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726140 ] ASF GitHub Bot logged work on HIVE-25895: - Author: ASF GitHub Bot Created on: 14/Feb/22 06:20 Start Date: 14/Feb/22 06:20 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #2980: URL: https://github.com/apache/hive/pull/2980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726140) Time Spent: 2h (was: 1h 50m) > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25907) IOW Directory queries fails to write data to final path when query result cache is enabled
[
https://issues.apache.org/jira/browse/HIVE-25907?focusedWorklogId=726118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726118
]
ASF GitHub Bot logged work on HIVE-25907:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 04:51
Start Date: 14/Feb/22 04:51
Worklog Time Spent: 10m
Work Description: shameersss1 removed a comment on pull request #2978:
URL: https://github.com/apache/hive/pull/2978#issuecomment-1023918982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726118)
Time Spent: 40m (was: 0.5h)
> IOW Directory queries fails to write data to final path when query result
> cache is enabled
> --
>
> Key: HIVE-25907
> URL: https://issues.apache.org/jira/browse/HIVE-25907
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> INSERT OVERWRITE DIRECTORY queries fails to write the data to the specified
> directory location when query result cache is enabled.
> *Steps to reproduce*
> {code:java}
> 1. create a data file with the following data
> 1 abc 10.5
> 2 def 11.5
> 2. create table pointing to that data
> create external table iowd(strct struct)
> row format delimited
> fields terminated by '\t'
> collection items terminated by ' '
> location '';
> 3. run the following query
> set hive.query.results.cache.enabled=true;
> INSERT OVERWRITE DIRECTORY "" SELECT * FROM iowd;
> {code}
> After execution of the above query, It is expected that the destination
> directory contains data from the table iowd, But due to HIVE-21386 it is not
> happening anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25907) IOW Directory queries fails to write data to final path when query result cache is enabled
[
https://issues.apache.org/jira/browse/HIVE-25907?focusedWorklogId=726119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726119
]
ASF GitHub Bot logged work on HIVE-25907:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 04:51
Start Date: 14/Feb/22 04:51
Worklog Time Spent: 10m
Work Description: shameersss1 commented on pull request #2978:
URL: https://github.com/apache/hive/pull/2978#issuecomment-1038639886
@kgyrtkirk - Can you please review the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726119)
Time Spent: 50m (was: 40m)
> IOW Directory queries fails to write data to final path when query result
> cache is enabled
> --
>
> Key: HIVE-25907
> URL: https://issues.apache.org/jira/browse/HIVE-25907
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> INSERT OVERWRITE DIRECTORY queries fails to write the data to the specified
> directory location when query result cache is enabled.
> *Steps to reproduce*
> {code:java}
> 1. create a data file with the following data
> 1 abc 10.5
> 2 def 11.5
> 2. create table pointing to that data
> create external table iowd(strct struct)
> row format delimited
> fields terminated by '\t'
> collection items terminated by ' '
> location '';
> 3. run the following query
> set hive.query.results.cache.enabled=true;
> INSERT OVERWRITE DIRECTORY "" SELECT * FROM iowd;
> {code}
> After execution of the above query, It is expected that the destination
> directory contains data from the table iowd, But due to HIVE-21386 it is not
> happening anymore.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-23556) Support hive.metastore.limit.partition.request for get_partitions_ps
[ https://issues.apache.org/jira/browse/HIVE-23556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491790#comment-17491790 ] iBenny commented on HIVE-23556: --- Since some parts of HiveMetastore.java have been moved into a file named HMSHandler.java, I am going to reopen a PR, > Support hive.metastore.limit.partition.request for get_partitions_ps > > > Key: HIVE-23556 > URL: https://issues.apache.org/jira/browse/HIVE-23556 > Project: Hive > Issue Type: Improvement >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-23556.2.patch, HIVE-23556.3.patch, > HIVE-23556.4.patch, HIVE-23556.patch > > > HIVE-13884 added the configuration hive.metastore.limit.partition.request to > limit the number of partitions that can be requested. > Currently, it takes in effect for the following MetaStore APIs > * get_partitions, > * get_partitions_with_auth, > * get_partitions_by_filter, > * get_partitions_spec_by_filter, > * get_partitions_by_expr, > but not for > * get_partitions_ps, > * get_partitions_ps_with_auth. > This issue proposes to apply the configuration also to get_partitions_ps and > get_partitions_ps_with_auth. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?focusedWorklogId=726111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726111 ] ASF GitHub Bot logged work on HIVE-25895: - Author: ASF GitHub Bot Created on: 14/Feb/22 04:11 Start Date: 14/Feb/22 04:11 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #2980: URL: https://github.com/apache/hive/pull/2980#discussion_r805492574 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java ## @@ -136,8 +142,9 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory, FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf); shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START) && fs.exists(failoverReadyMarker); - isFailover = - checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, OptimisedBootstrapUtils.EVENT_ACK_FILE); + isFirstFailover = checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, EVENT_ACK_FILE); Review comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726111) Time Spent: 1h 50m (was: 1h 40m) > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?focusedWorklogId=726109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726109
]
ASF GitHub Bot logged work on HIVE-25912:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 04:07
Start Date: 14/Feb/22 04:07
Worklog Time Spent: 10m
Work Description: baifachuan commented on pull request #3009:
URL: https://github.com/apache/hive/pull/3009#issuecomment-1038613768
> +1 pending tests
Hi pvary, all the tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726109)
Remaining Estimate: 79h 10m (was: 79h 20m)
Time Spent: 16h 50m (was: 16h 40m)
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2, 4.0.0
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Attachments: hive bugs.png, hive-bug-01.png
>
> Original Estimate: 96h
> Time Spent: 16h 50m
> Remaining Estimate: 79h 10m
>
> *new update:*
> I test the master branch, have the same problem.
> --
> ENV:
> Hive 3.1.2
> HDFS:3.3.1
> enable OpenLDAP and Ranger .
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24545) jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/HIVE-24545?focusedWorklogId=726068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726068
]
ASF GitHub Bot logged work on HIVE-24545:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:39
Start Date: 14/Feb/22 02:39
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1789:
URL: https://github.com/apache/hive/pull/1789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726068)
Time Spent: 1h 40m (was: 1.5h)
> jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
>
>
> Key: HIVE-24545
> URL: https://issues.apache.org/jira/browse/HIVE-24545
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> I found this while IOW on TPCDS 10TB:
> {code}
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 4210 421000
> 0 362
> Reducer 2 .. llap SUCCEEDED10110100
> 0 2
> Reducer 3 .. llap SUCCEEDED 1009 100900
> 0 1
> --
> VERTICES: 03/03 [==>>] 100% ELAPSED TIME: 12613.62 s
> --
> 20/12/16 01:37:36 [main]: WARN jdbc.HiveStatement: Number of rows is greater
> than Integer.MAX_VALUE
> {code}
> my scenario was:
> {code}
> set hive.exec.max.dynamic.partitions=2000;
> drop table if exists test_sales_2;
> create table test_sales_2 like
> tpcds_bin_partitioned_acid_orc_1.store_sales;
> insert overwrite table test_sales_2 select * from
> tpcds_bin_partitioned_acid_orc_1.store_sales where ss_sold_date_sk >
> 2451868;
> {code}
> regarding affected row numbers:
> {code}
> select count(*) from tpcds_bin_partitioned_acid_orc_1.store_sales where
> ss_sold_date_sk > 2451868;
> +--+
> | _c0 |
> +--+
> | 12287871907 |
> +--+
> {code}
> I guess we should switch to long
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25793) Isolate metastore metrics related to a retrying handler
[ https://issues.apache.org/jira/browse/HIVE-25793?focusedWorklogId=726042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726042 ] ASF GitHub Bot logged work on HIVE-25793: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:37 Start Date: 14/Feb/22 02:37 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2860: URL: https://github.com/apache/hive/pull/2860#issuecomment-1037602287 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726042) Time Spent: 1h 50m (was: 1h 40m) > Isolate metastore metrics related to a retrying handler > --- > > Key: HIVE-25793 > URL: https://issues.apache.org/jira/browse/HIVE-25793 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.8, 3.1.2, 4.0.0 >Reporter: Jeongdae Kim >Assignee: Jeongdae Kim >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > We can see that a metastore api count increments twice for one RPC, because a > metastore server makes two timers with same name for one RPC. one for a > retrying proxy handler and another for a hms handler. It will make api > metrics more inaccurate, especially during retrying. > I think new metrics for retrying proxy will be helpful to make each metric > accurate. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=726028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726028 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:36 Start Date: 14/Feb/22 02:36 Worklog Time Spent: 10m Work Description: shameersss1 commented on pull request #3008: URL: https://github.com/apache/hive/pull/3008#issuecomment-1038029216 @kgyrtkirk - Finally a green run. Can we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726028) Time Spent: 2h 20m (was: 2h 10m) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.1
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=726023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726023 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:36 Start Date: 14/Feb/22 02:36 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1742: URL: https://github.com/apache/hive/pull/1742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 726023) Time Spent: 8.05h (was: 7h 53m) > Upgrade Hadoop to 3.3.1 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 8.05h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25576) Add config to parse date with older date format
[
https://issues.apache.org/jira/browse/HIVE-25576?focusedWorklogId=726001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726001
]
ASF GitHub Bot logged work on HIVE-25576:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:34
Start Date: 14/Feb/22 02:34
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2690:
URL: https://github.com/apache/hive/pull/2690#issuecomment-1037602311
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 726001)
Time Spent: 2.5h (was: 2h 20m)
> Add config to parse date with older date format
> ---
>
> Key: HIVE-25576
> URL: https://issues.apache.org/jira/browse/HIVE-25576
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.1.0, 3.0.0, 3.1.1, 3.1.2, 4.0.0
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> *History*
> *Hive 1.2* -
> VM time zone set to Asia/Bangkok
> *Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
> UTC','-MM-dd HH:mm:ss z'));
> *Result* - 1800-01-01 07:00:00
> *Implementation details* -
> SimpleDateFormat formatter = new SimpleDateFormat(pattern);
> Long unixtime = formatter.parse(textval).getTime() / 1000;
> Date date = new Date(unixtime * 1000L);
> https://docs.oracle.com/javase/8/docs/api/java/util/Date.html . In official
> documentation they have mention that "Unfortunately, the API for these
> functions was not amenable to internationalization and The corresponding
> methods in Date are deprecated" . Due to that this is producing wrong result
> *Master branch* -
> set hive.local.time.zone=Asia/Bangkok;
> *Query* - SELECT FROM_UNIXTIME(UNIX_TIMESTAMP('1800-01-01 00:00:00
> UTC','-MM-dd HH:mm:ss z'));
> *Result* - 1800-01-01 06:42:04
> *Implementation details* -
> DateTimeFormatter dtformatter = new DateTimeFormatterBuilder()
> .parseCaseInsensitive()
> .appendPattern(pattern)
> .toFormatter();
> ZonedDateTime zonedDateTime =
> ZonedDateTime.parse(textval,dtformatter).withZoneSameInstant(ZoneId.of(timezone));
> Long dttime = zonedDateTime.toInstant().getEpochSecond();
> *Problem*-
> Now *SimpleDateFormat* has been replaced with *DateTimeFormatter* which is
> giving the correct result but it is not backword compatible. Which is causing
> issue at time for migration to new version. Because the older data written is
> using Hive 1.x or 2.x is not compatible with *DateTimeFormatter*.
> *Solution*
> Introduce an config "hive.legacy.timeParserPolicy" with following values -
> 1. *True*- use *SimpleDateFormat*
> 2. *False* - use *DateTimeFormatter*
> Note: apache spark also face the same issue
> https://issues.apache.org/jira/browse/SPARK-30668
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24289) RetryingMetaStoreClient should not retry connecting to HMS on genuine errors
[
https://issues.apache.org/jira/browse/HIVE-24289?focusedWorklogId=725962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725962
]
ASF GitHub Bot logged work on HIVE-24289:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 02:30
Start Date: 14/Feb/22 02:30
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2675:
URL: https://github.com/apache/hive/pull/2675#issuecomment-1037602325
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 725962)
Time Spent: 40m (was: 0.5h)
> RetryingMetaStoreClient should not retry connecting to HMS on genuine errors
>
>
> Key: HIVE-24289
> URL: https://issues.apache.org/jira/browse/HIVE-24289
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When there is genuine error from HMS, it should not be retried in
> RetryingMetaStoreClient.
> For e.g, following query would be retried multiple times (~20+ times) in HMS
> causing huge delay in processing, even though this constraint is available in
> HMS.
> It should just throw exception to client and stop retrying in such cases.
> {noformat}
> alter table web_sales add constraint tpcds_bin_partitioned_orc_1_ws_s_hd
> foreign key (ws_ship_hdemo_sk) references household_demographics
> (hd_demo_sk) disable novalidate rely;
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: Internal error processing
> add_foreign_key
> at org.apache.hadoop.hive.ql.metadata.Hive.addForeignKey(Hive.java:5914)
> ..
> ...
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> add_foreign_key
>at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
>at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_foreign_key(ThriftHiveMetastore.java:1872)
> {noformat}
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java#L256
> For e.g, if exception contains "Internal error processing ", it could stop
> retrying all over again.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=725937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725937 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:29 Start Date: 14/Feb/22 02:29 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #3008: URL: https://github.com/apache/hive/pull/3008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725937) Time Spent: 2h 10m (was: 2h) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25808) Analyse table does not fail for non existing partitions
[ https://issues.apache.org/jira/browse/HIVE-25808?focusedWorklogId=725946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725946 ] ASF GitHub Bot logged work on HIVE-25808: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:29 Start Date: 14/Feb/22 02:29 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2875: URL: https://github.com/apache/hive/pull/2875#issuecomment-1038489509 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725946) Time Spent: 0.5h (was: 20m) > Analyse table does not fail for non existing partitions > --- > > Key: HIVE-25808 > URL: https://issues.apache.org/jira/browse/HIVE-25808 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > If all the column names are given in the analyse command , then the query > fails. But if all the partition column values are not given then its not > failing. > analyze table tbl partition *(fld1 = 2, fld2 = 3)* COMPUTE STATISTICS FOR > COLUMNS – This will fail with SemanticException, if partition corresponds to > fld1 = 2, fld2 = 3 does not exists. But analyze table tbl partition *(fld1 = > 2)* COMPUTE STATISTICS FOR COLUMNS, this will not fail and it will compute > stats for whole table. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=725927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725927 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:28 Start Date: 14/Feb/22 02:28 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #3008: URL: https://github.com/apache/hive/pull/3008#issuecomment-1038038083 @shameersss1 ; yeah - its particularily annoying when failures like this happen when it doesnt seem related at all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725927) Time Spent: 2h (was: 1h 50m) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25750) Beeline: Creating a standalone tarball by isolating dependencies
[ https://issues.apache.org/jira/browse/HIVE-25750?focusedWorklogId=725921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725921 ] ASF GitHub Bot logged work on HIVE-25750: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:28 Start Date: 14/Feb/22 02:28 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2824: URL: https://github.com/apache/hive/pull/2824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725921) Time Spent: 1.5h (was: 1h 20m) > Beeline: Creating a standalone tarball by isolating dependencies > > > Key: HIVE-25750 > URL: https://issues.apache.org/jira/browse/HIVE-25750 > Project: Hive > Issue Type: Bug >Reporter: Abhay >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The code to create a standalone beeline tarball was created as part of this > ticket https://issues.apache.org/jira/browse/HIVE-24348. However, a bug was > reported in the case when the beeline is tried to install without the hadoop > installed. > The beeline script complains of missing dependencies when it is run. > The ask as part of this ticket is to fix that bug. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25789) Replication metrics and logs show wrong repl id when no of events replicated is 0
[ https://issues.apache.org/jira/browse/HIVE-25789?focusedWorklogId=725907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725907 ] ASF GitHub Bot logged work on HIVE-25789: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:27 Start Date: 14/Feb/22 02:27 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2854: URL: https://github.com/apache/hive/pull/2854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725907) Time Spent: 0.5h (was: 20m) > Replication metrics and logs show wrong repl id when no of events replicated > is 0 > - > > Key: HIVE-25789 > URL: https://issues.apache.org/jira/browse/HIVE-25789 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > When no of events replicated in incremental cycle is 0, logs and metrics show > wrong value of lastReplId. REPL STATUS command still gives the right value. > Logs show a value of 'null'. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25335) Unreasonable setting reduce number, when join big size table(but small row count) and small size table
[ https://issues.apache.org/jira/browse/HIVE-25335?focusedWorklogId=725903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725903 ] ASF GitHub Bot logged work on HIVE-25335: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:27 Start Date: 14/Feb/22 02:27 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2490: URL: https://github.com/apache/hive/pull/2490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725903) Time Spent: 2h (was: 1h 50m) > Unreasonable setting reduce number, when join big size table(but small row > count) and small size table > -- > > Key: HIVE-25335 > URL: https://issues.apache.org/jira/browse/HIVE-25335 > Project: Hive > Issue Type: Improvement >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25335.001.patch > > Time Spent: 2h > Remaining Estimate: 0h > > I found an application which is slow in our cluster, because the proccess > bytes of one reduce is very huge, but only two reduce. > when I debug, I found the reason. Because in this sql, one big size table > (about 30G) with few row count(about 3.5M), another small size table (about > 100M) have more row count (about 3.6M). So JoinStatsRule.process only use > 100M to estimate reducer's number. But we need to process 30G byte in fact. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?focusedWorklogId=725901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725901 ] ASF GitHub Bot logged work on HIVE-25895: - Author: ASF GitHub Bot Created on: 14/Feb/22 02:26 Start Date: 14/Feb/22 02:26 Worklog Time Spent: 10m Work Description: pkumarsinha commented on a change in pull request #2980: URL: https://github.com/apache/hive/pull/2980#discussion_r805413504 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java ## @@ -136,8 +142,9 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory, FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf); shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START) && fs.exists(failoverReadyMarker); - isFailover = - checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, OptimisedBootstrapUtils.EVENT_ACK_FILE); + isFirstFailover = checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, EVENT_ACK_FILE); Review comment: nit: Could create and re-use the Path object. e.g Path dumpDirParent = new Path(dumpDirectory).getParent(); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725901) Time Spent: 1h 40m (was: 1.5h) > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24545) jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/HIVE-24545?focusedWorklogId=725862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725862
]
ASF GitHub Bot logged work on HIVE-24545:
-
Author: ASF GitHub Bot
Created on: 14/Feb/22 00:13
Start Date: 14/Feb/22 00:13
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1789:
URL: https://github.com/apache/hive/pull/1789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 725862)
Time Spent: 1.5h (was: 1h 20m)
> jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
>
>
> Key: HIVE-24545
> URL: https://issues.apache.org/jira/browse/HIVE-24545
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> I found this while IOW on TPCDS 10TB:
> {code}
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 4210 421000
> 0 362
> Reducer 2 .. llap SUCCEEDED10110100
> 0 2
> Reducer 3 .. llap SUCCEEDED 1009 100900
> 0 1
> --
> VERTICES: 03/03 [==>>] 100% ELAPSED TIME: 12613.62 s
> --
> 20/12/16 01:37:36 [main]: WARN jdbc.HiveStatement: Number of rows is greater
> than Integer.MAX_VALUE
> {code}
> my scenario was:
> {code}
> set hive.exec.max.dynamic.partitions=2000;
> drop table if exists test_sales_2;
> create table test_sales_2 like
> tpcds_bin_partitioned_acid_orc_1.store_sales;
> insert overwrite table test_sales_2 select * from
> tpcds_bin_partitioned_acid_orc_1.store_sales where ss_sold_date_sk >
> 2451868;
> {code}
> regarding affected row numbers:
> {code}
> select count(*) from tpcds_bin_partitioned_acid_orc_1.store_sales where
> ss_sold_date_sk > 2451868;
> +--+
> | _c0 |
> +--+
> | 12287871907 |
> +--+
> {code}
> I guess we should switch to long
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.1
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=725861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725861 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 14/Feb/22 00:13 Start Date: 14/Feb/22 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1742: URL: https://github.com/apache/hive/pull/1742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725861) Time Spent: 7h 53m (was: 7h 43m) > Upgrade Hadoop to 3.3.1 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 7h 53m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25789) Replication metrics and logs show wrong repl id when no of events replicated is 0
[ https://issues.apache.org/jira/browse/HIVE-25789?focusedWorklogId=725859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725859 ] ASF GitHub Bot logged work on HIVE-25789: - Author: ASF GitHub Bot Created on: 14/Feb/22 00:13 Start Date: 14/Feb/22 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2854: URL: https://github.com/apache/hive/pull/2854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725859) Time Spent: 20m (was: 10m) > Replication metrics and logs show wrong repl id when no of events replicated > is 0 > - > > Key: HIVE-25789 > URL: https://issues.apache.org/jira/browse/HIVE-25789 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When no of events replicated in incremental cycle is 0, logs and metrics show > wrong value of lastReplId. REPL STATUS command still gives the right value. > Logs show a value of 'null'. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25335) Unreasonable setting reduce number, when join big size table(but small row count) and small size table
[ https://issues.apache.org/jira/browse/HIVE-25335?focusedWorklogId=725860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725860 ] ASF GitHub Bot logged work on HIVE-25335: - Author: ASF GitHub Bot Created on: 14/Feb/22 00:13 Start Date: 14/Feb/22 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2490: URL: https://github.com/apache/hive/pull/2490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725860) Time Spent: 1h 50m (was: 1h 40m) > Unreasonable setting reduce number, when join big size table(but small row > count) and small size table > -- > > Key: HIVE-25335 > URL: https://issues.apache.org/jira/browse/HIVE-25335 > Project: Hive > Issue Type: Improvement >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25335.001.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > I found an application which is slow in our cluster, because the proccess > bytes of one reduce is very huge, but only two reduce. > when I debug, I found the reason. Because in this sql, one big size table > (about 30G) with few row count(about 3.5M), another small size table (about > 100M) have more row count (about 3.6M). So JoinStatsRule.process only use > 100M to estimate reducer's number. But we need to process 30G byte in fact. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25808) Analyse table does not fail for non existing partitions
[ https://issues.apache.org/jira/browse/HIVE-25808?focusedWorklogId=725858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725858 ] ASF GitHub Bot logged work on HIVE-25808: - Author: ASF GitHub Bot Created on: 14/Feb/22 00:13 Start Date: 14/Feb/22 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2875: URL: https://github.com/apache/hive/pull/2875#issuecomment-1038489509 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725858) Time Spent: 20m (was: 10m) > Analyse table does not fail for non existing partitions > --- > > Key: HIVE-25808 > URL: https://issues.apache.org/jira/browse/HIVE-25808 > Project: Hive > Issue Type: Bug >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > If all the column names are given in the analyse command , then the query > fails. But if all the partition column values are not given then its not > failing. > analyze table tbl partition *(fld1 = 2, fld2 = 3)* COMPUTE STATISTICS FOR > COLUMNS – This will fail with SemanticException, if partition corresponds to > fld1 = 2, fld2 = 3 does not exists. But analyze table tbl partition *(fld1 = > 2)* COMPUTE STATISTICS FOR COLUMNS, this will not fail and it will compute > stats for whole table. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25895) Bootstrap tables in table_diff during Incremental Load
[ https://issues.apache.org/jira/browse/HIVE-25895?focusedWorklogId=725841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725841 ] ASF GitHub Bot logged work on HIVE-25895: - Author: ASF GitHub Bot Created on: 13/Feb/22 20:52 Start Date: 13/Feb/22 20:52 Worklog Time Spent: 10m Work Description: pkumarsinha commented on a change in pull request #2980: URL: https://github.com/apache/hive/pull/2980#discussion_r805413504 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java ## @@ -136,8 +142,9 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory, FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf); shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START) && fs.exists(failoverReadyMarker); - isFailover = - checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, OptimisedBootstrapUtils.EVENT_ACK_FILE); + isFirstFailover = checkFileExists(new Path(dumpDirectory).getParent(), hiveConf, EVENT_ACK_FILE); Review comment: nit: Could create and re-use the Path object. e.g Path dumpDirParent = new Path(dumpDirectory).getParent(); -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725841) Time Spent: 1.5h (was: 1h 20m) > Bootstrap tables in table_diff during Incremental Load > -- > > Key: HIVE-25895 > URL: https://issues.apache.org/jira/browse/HIVE-25895 > Project: Hive > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Consume the table_diff_ack file and do a bootstrap dump & load for those > tables -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-25942. - Resolution: Fixed merged into master. Thank you [~srahman]! > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=725779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725779 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 13/Feb/22 11:32 Start Date: 13/Feb/22 11:32 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #3008: URL: https://github.com/apache/hive/pull/3008#issuecomment-1038038083 @shameersss1 ; yeah - its particularily annoying when failures like this happen when it doesnt seem related at all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725779) Time Spent: 1h 50m (was: 1h 40m) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=725778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725778 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 13/Feb/22 11:31 Start Date: 13/Feb/22 11:31 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #3008: URL: https://github.com/apache/hive/pull/3008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725778) Time Spent: 1h 40m (was: 1.5h) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25942) Upgrade commons-io to 2.8.0 due to CVE-2021-29425
[ https://issues.apache.org/jira/browse/HIVE-25942?focusedWorklogId=725777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-725777 ] ASF GitHub Bot logged work on HIVE-25942: - Author: ASF GitHub Bot Created on: 13/Feb/22 11:19 Start Date: 13/Feb/22 11:19 Worklog Time Spent: 10m Work Description: shameersss1 commented on pull request #3008: URL: https://github.com/apache/hive/pull/3008#issuecomment-1038029216 @kgyrtkirk - Finally a green run. Can we merge this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 725777) Time Spent: 1.5h (was: 1h 20m) > Upgrade commons-io to 2.8.0 due to CVE-2021-29425 > - > > Key: HIVE-25942 > URL: https://issues.apache.org/jira/browse/HIVE-25942 > Project: Hive > Issue Type: Bug >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Due to [CVE-2021-29425|https://nvd.nist.gov/vuln/detail/CVE-2021-29425] all > the commons-io versions below 2.7 are affected. > Tez and Hadoop have upgraded commons-io to 2.8.0 in > [TEZ-4353|https://issues.apache.org/jira/browse/TEZ-4353] and > [HADOOP-17683|https://issues.apache.org/jira/browse/HADOOP-17683] > respectively and it will be good if Hive also follows the same. -- This message was sent by Atlassian Jira (v8.20.1#820001)
