[jira] [Work logged] (HIVE-25497) Bump ORC to 1.7.1
[ https://issues.apache.org/jira/browse/HIVE-25497?focusedWorklogId=692238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692238 ] ASF GitHub Bot logged work on HIVE-25497: - Author: ASF GitHub Bot Created on: 08/Dec/21 05:34 Start Date: 08/Dec/21 05:34 Worklog Time Spent: 10m Work Description: pgaref opened a new pull request #2853: URL: https://github.com/apache/hive/pull/2853 Bumping to 1.7.1. LLAP EncodedTreeReaderFactory has a hard implementing its own TreeReaderFactory but initially we are going to avoid LAZY IO here as everything is kept in memory. ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692238) Time Spent: 1.5h (was: 1h 20m) > Bump ORC to 1.7.1 > - > > Key: HIVE-25497 > URL: https://issues.apache.org/jira/browse/HIVE-25497 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: William Hyun >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25497) Bump ORC to 1.7.1
[ https://issues.apache.org/jira/browse/HIVE-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-25497: -- Summary: Bump ORC to 1.7.1 (was: Bump ORC to 1.7.0) > Bump ORC to 1.7.1 > - > > Key: HIVE-25497 > URL: https://issues.apache.org/jira/browse/HIVE-25497 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: William Hyun >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25497) Bump ORC to 1.7.1
[ https://issues.apache.org/jira/browse/HIVE-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-25497: - Assignee: Panagiotis Garefalakis > Bump ORC to 1.7.1 > - > > Key: HIVE-25497 > URL: https://issues.apache.org/jira/browse/HIVE-25497 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: William Hyun >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25367) Fix TestReplicationScenariosAcidTables#testMultiDBTxn
[
https://issues.apache.org/jira/browse/HIVE-25367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pravin Sinha updated HIVE-25367:
Description:
[http://ci.hive.apache.org/job/hive-flaky-check/331]
[http://ci.hive.apache.org/job/hive-flaky-check/332]
CC: [~aasha]
{color:#172b4d}Fix concurrency issue in TaskRunner
{color}
was:
[http://ci.hive.apache.org/job/hive-flaky-check/331]
[http://ci.hive.apache.org/job/hive-flaky-check/332]
CC: [~aasha]
> Fix TestReplicationScenariosAcidTables#testMultiDBTxn
> -
>
> Key: HIVE-25367
> URL: https://issues.apache.org/jira/browse/HIVE-25367
> Project: Hive
> Issue Type: Test
> Components: repl
>Reporter: Peter Vary
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> [http://ci.hive.apache.org/job/hive-flaky-check/331]
> [http://ci.hive.apache.org/job/hive-flaky-check/332]
> CC: [~aasha]
>
> {color:#172b4d}Fix concurrency issue in TaskRunner
> {color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25335) Unreasonable setting reduce number, when join big size table(but small row count) and small size table
[ https://issues.apache.org/jira/browse/HIVE-25335?focusedWorklogId=692166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692166 ] ASF GitHub Bot logged work on HIVE-25335: - Author: ASF GitHub Bot Created on: 08/Dec/21 00:13 Start Date: 08/Dec/21 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2490: URL: https://github.com/apache/hive/pull/2490#issuecomment-988363846 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692166) Time Spent: 1h 20m (was: 1h 10m) > Unreasonable setting reduce number, when join big size table(but small row > count) and small size table > -- > > Key: HIVE-25335 > URL: https://issues.apache.org/jira/browse/HIVE-25335 > Project: Hive > Issue Type: Improvement >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25335.001.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I found an application which is slow in our cluster, because the proccess > bytes of one reduce is very huge, but only two reduce. > when I debug, I found the reason. Because in this sql, one big size table > (about 30G) with few row count(about 3.5M), another small size table (about > 100M) have more row count (about 3.6M). So JoinStatsRule.process only use > 100M to estimate reducer's number. But we need to process 30G byte in fact. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25514) Alter table with partitions should honor {OWNER} policies from Apache Ranger in the HMS
[
https://issues.apache.org/jira/browse/HIVE-25514?focusedWorklogId=692164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692164
]
ASF GitHub Bot logged work on HIVE-25514:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 00:13
Start Date: 08/Dec/21 00:13
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #2634:
URL: https://github.com/apache/hive/pull/2634#issuecomment-988363801
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692164)
Time Spent: 1.5h (was: 1h 20m)
> Alter table with partitions should honor {OWNER} policies from Apache Ranger
> in the HMS
> ---
>
> Key: HIVE-25514
> URL: https://issues.apache.org/jira/browse/HIVE-25514
> Project: Hive
> Issue Type: Bug
> Components: Hive, Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The following commands should honor \{OWNER} policies from Apache Ranger in
> the HMS.
> {code:java}
> Show partitions table_name;
> alter table foo.table_name partition (country='us') rename to partition
> (country='canada);
> alter table foo.table_name drop partition (id='canada');{code}
> The examples above are tables with partitions. So the partition APIs in HMS
> should be modifed to honor \{owner} policies from Apache ranger.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25540) Enable batch update of column stats only for MySql and Postgres
[ https://issues.apache.org/jira/browse/HIVE-25540?focusedWorklogId=692165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692165 ] ASF GitHub Bot logged work on HIVE-25540: - Author: ASF GitHub Bot Created on: 08/Dec/21 00:13 Start Date: 08/Dec/21 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2657: URL: https://github.com/apache/hive/pull/2657#issuecomment-988363765 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692165) Time Spent: 40m (was: 0.5h) > Enable batch update of column stats only for MySql and Postgres > > > Key: HIVE-25540 > URL: https://issues.apache.org/jira/browse/HIVE-25540 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The batch updation of partition column stats using direct sql is tested only > for MySql and Postgres. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25786) Auto-close browser window/tab after successful auth with SSO(SAML) authentication
[ https://issues.apache.org/jira/browse/HIVE-25786?focusedWorklogId=692053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692053 ] ASF GitHub Bot logged work on HIVE-25786: - Author: ASF GitHub Bot Created on: 07/Dec/21 20:43 Start Date: 07/Dec/21 20:43 Worklog Time Spent: 10m Work Description: saihemanth-cloudera opened a new pull request #2851: URL: https://github.com/apache/hive/pull/2851 ### What changes were proposed in this pull request? Added html/JS script to auto close browser/tab after successful auth with SSO. ### Why are the changes needed? Eases the user interaction where previously user had to manually close the browser tab on successful auth. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Local machine, Remote cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692053) Remaining Estimate: 0h Time Spent: 10m > Auto-close browser window/tab after successful auth with SSO(SAML) > authentication > - > > Key: HIVE-25786 > URL: https://issues.apache.org/jira/browse/HIVE-25786 > Project: Hive > Issue Type: Bug > Components: Authentication, HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When a client(beeline/DBeaver) tries to connect to HS2. Each connection opens > a new tab which is a bit of an annoying user experience. This task is to > Auto-close the window on a successful auth after a timed interval (5 sec). On > an unsuccessful auth, the browser window would not be auto-closed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25786) Auto-close browser window/tab after successful auth with SSO(SAML) authentication
[ https://issues.apache.org/jira/browse/HIVE-25786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25786: -- Labels: pull-request-available (was: ) > Auto-close browser window/tab after successful auth with SSO(SAML) > authentication > - > > Key: HIVE-25786 > URL: https://issues.apache.org/jira/browse/HIVE-25786 > Project: Hive > Issue Type: Bug > Components: Authentication, HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When a client(beeline/DBeaver) tries to connect to HS2. Each connection opens > a new tab which is a bit of an annoying user experience. This task is to > Auto-close the window on a successful auth after a timed interval (5 sec). On > an unsuccessful auth, the browser window would not be auto-closed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25786) Auto-close browser window/tab after successful auth with SSO(SAML) authentication
[ https://issues.apache.org/jira/browse/HIVE-25786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-25786: > Auto-close browser window/tab after successful auth with SSO(SAML) > authentication > - > > Key: HIVE-25786 > URL: https://issues.apache.org/jira/browse/HIVE-25786 > Project: Hive > Issue Type: Bug > Components: Authentication, HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > When a client(beeline/DBeaver) tries to connect to HS2. Each connection opens > a new tab which is a bit of an annoying user experience. This task is to > Auto-close the window on a successful auth after a timed interval (5 sec). On > an unsuccessful auth, the browser window would not be auto-closed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25785) Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690
[ https://issues.apache.org/jira/browse/HIVE-25785?focusedWorklogId=691956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691956 ] ASF GitHub Bot logged work on HIVE-25785: - Author: ASF GitHub Bot Created on: 07/Dec/21 18:35 Start Date: 07/Dec/21 18:35 Worklog Time Spent: 10m Work Description: saihemanth-cloudera opened a new pull request #2850: URL: https://github.com/apache/hive/pull/2850 ### What changes were proposed in this pull request? Version upgrade for xmlsec in the pom ### Why are the changes needed? To avoid CVE-2021-40690 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Local machine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691956) Remaining Estimate: 0h Time Spent: 10m > Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690 > --- > > Key: HIVE-25785 > URL: https://issues.apache.org/jira/browse/HIVE-25785 > Project: Hive > Issue Type: Bug > Components: Hive, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hive is currently pulling in xmlsec 2.2.1. This needs to be upgraded to > 2.2.3+ due to CVE-2021-40690. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25785) Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690
[ https://issues.apache.org/jira/browse/HIVE-25785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25785: -- Labels: pull-request-available (was: ) > Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690 > --- > > Key: HIVE-25785 > URL: https://issues.apache.org/jira/browse/HIVE-25785 > Project: Hive > Issue Type: Bug > Components: Hive, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Hive is currently pulling in xmlsec 2.2.1. This needs to be upgraded to > 2.2.3+ due to CVE-2021-40690. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25785) Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690
[ https://issues.apache.org/jira/browse/HIVE-25785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-25785: > Upgrade xmlsec to 2.1.7/2.2.3 due to CVE-2021-40690 > --- > > Key: HIVE-25785 > URL: https://issues.apache.org/jira/browse/HIVE-25785 > Project: Hive > Issue Type: Bug > Components: Hive, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > Hive is currently pulling in xmlsec 2.2.1. This needs to be upgraded to > 2.2.3+ due to CVE-2021-40690. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25780) DistinctExpansion creates more than 64 grouping sets II
[ https://issues.apache.org/jira/browse/HIVE-25780?focusedWorklogId=691925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691925 ] ASF GitHub Bot logged work on HIVE-25780: - Author: ASF GitHub Bot Created on: 07/Dec/21 18:04 Start Date: 07/Dec/21 18:04 Worklog Time Spent: 10m Work Description: kgyrtkirk opened a new pull request #2849: URL: https://github.com/apache/hive/pull/2849 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691925) Remaining Estimate: 0h Time Spent: 10m > DistinctExpansion creates more than 64 grouping sets II > --- > > Key: HIVE-25780 > URL: https://issues.apache.org/jira/browse/HIVE-25780 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-25498 have fixed this when there are only count(distinct x) queries. > however after the rewrite happens grouping sets are used to handle group by > columns as well -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25780) DistinctExpansion creates more than 64 grouping sets II
[ https://issues.apache.org/jira/browse/HIVE-25780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25780: -- Labels: pull-request-available (was: ) > DistinctExpansion creates more than 64 grouping sets II > --- > > Key: HIVE-25780 > URL: https://issues.apache.org/jira/browse/HIVE-25780 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > HIVE-25498 have fixed this when there are only count(distinct x) queries. > however after the rewrite happens grouping sets are used to handle group by > columns as well -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[ https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=691879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691879 ] ASF GitHub Bot logged work on HIVE-25755: - Author: ASF GitHub Bot Created on: 07/Dec/21 17:08 Start Date: 07/Dec/21 17:08 Worklog Time Spent: 10m Work Description: deniskuzZ merged pull request #2831: URL: https://github.com/apache/hive/pull/2831 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691879) Time Spent: 2h 10m (was: 2h) > TestTxnCommands classes refactor > > > Key: HIVE-25755 > URL: https://issues.apache.org/jira/browse/HIVE-25755 > Project: Hive > Issue Type: Test >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25784) Upgrade Arrow version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-25784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao reassigned HIVE-25784: -- > Upgrade Arrow version to 2.0.0 > -- > > Key: HIVE-25784 > URL: https://issues.apache.org/jira/browse/HIVE-25784 > Project: Hive > Issue Type: Task >Affects Versions: 4.0.0 >Reporter: Adesh Kumar Rao >Assignee: Adesh Kumar Rao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25777) ACID: Pick the compactor transaction over insert dir
[
https://issues.apache.org/jira/browse/HIVE-25777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-25777:
--
Labels: Compaction (was: )
> ACID: Pick the compactor transaction over insert dir
>
>
> Key: HIVE-25777
> URL: https://issues.apache.org/jira/browse/HIVE-25777
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Gopal Vijayaraghavan
>Priority: Major
> Labels: Compaction
>
> If there are two competing versions of a particular write-id, one from the
> compactor and another from the original insert, always pick the compactor one
> once it is committed.
> If the directory structure looks like
> {code}
> base_11/
> base_11_v192/
> {code}
> Then always pick the v192 transaction if txnid=192 is committed.
> This is required to ensure that the raw base_ dir can be deleted safely on
> non-atomic directory deletions (like s3), without a race condition between
> getSplits and the actual file-reader.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25751) Ignore exceptions related to interruption when the limit is reached
[
https://issues.apache.org/jira/browse/HIVE-25751?focusedWorklogId=691759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691759
]
ASF GitHub Bot logged work on HIVE-25751:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 13:51
Start Date: 07/Dec/21 13:51
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #2828:
URL: https://github.com/apache/hive/pull/2828#discussion_r764012497
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
##
@@ -138,10 +142,28 @@
rr.start();
return result;
} catch (Exception ex) {
- throw new IOException(ex);
+ Throwable rootCause = JavaUtils.findRootCause(ex);
Review comment:
good catch @rbalamohan , easily reproduced with tez driver, added a fix
for that here:
before fix (task attempt failure):
https://issues.apache.org/jira/secure/attachment/13037081/syslog
after fix (task attempt success):
https://issues.apache.org/jira/secure/attachment/13037083/syslog_with_fix
please find fix in last commit, let me know if your +1 still holds, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691759)
Time Spent: 1h 10m (was: 1h)
> Ignore exceptions related to interruption when the limit is reached
> ---
>
> Key: HIVE-25751
> URL: https://issues.apache.org/jira/browse/HIVE-25751
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: syslog, syslog_with_fix
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The problem is well described on TEZ-4356, but it will be handled from hive
> instead.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25751) Ignore exceptions related to interruption when the limit is reached
[ https://issues.apache.org/jira/browse/HIVE-25751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-25751: Attachment: syslog_with_fix > Ignore exceptions related to interruption when the limit is reached > --- > > Key: HIVE-25751 > URL: https://issues.apache.org/jira/browse/HIVE-25751 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Attachments: syslog, syslog_with_fix > > Time Spent: 1h > Remaining Estimate: 0h > > The problem is well described on TEZ-4356, but it will be handled from hive > instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25774) Add ASF license for newly created files in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25774?focusedWorklogId=691748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691748 ] ASF GitHub Bot logged work on HIVE-25774: - Author: ASF GitHub Bot Created on: 07/Dec/21 13:34 Start Date: 07/Dec/21 13:34 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #2844: URL: https://github.com/apache/hive/pull/2844#issuecomment-987931015 > Pushed to master @dengzhhu653! It might worth to add the rat check to the CI, so we will not face this issue again. Is it possible to do it? > > Thanks, Peter Thank you for the merge. Agreed, I will open another jira for this pre check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691748) Time Spent: 1h 10m (was: 1h) > Add ASF license for newly created files in standalone-metastore > --- > > Key: HIVE-25774 > URL: https://issues.apache.org/jira/browse/HIVE-25774 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25751) Ignore exceptions related to interruption when the limit is reached
[ https://issues.apache.org/jira/browse/HIVE-25751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-25751: Attachment: syslog > Ignore exceptions related to interruption when the limit is reached > --- > > Key: HIVE-25751 > URL: https://issues.apache.org/jira/browse/HIVE-25751 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Attachments: syslog > > Time Spent: 1h > Remaining Estimate: 0h > > The problem is well described on TEZ-4356, but it will be handled from hive > instead. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25774) Add ASF license for newly created files in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25774?focusedWorklogId=691740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691740 ] ASF GitHub Bot logged work on HIVE-25774: - Author: ASF GitHub Bot Created on: 07/Dec/21 13:19 Start Date: 07/Dec/21 13:19 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2844: URL: https://github.com/apache/hive/pull/2844#issuecomment-987918694 Pushed to master @dengzhhu653! It might worth to add the rat check to the CI, so we will not face this issue again. Is it possible to do it? Thanks, Peter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691740) Time Spent: 1h (was: 50m) > Add ASF license for newly created files in standalone-metastore > --- > > Key: HIVE-25774 > URL: https://issues.apache.org/jira/browse/HIVE-25774 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25774) Add ASF license for newly created files in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-25774. --- Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the patch [~dengzh]! > Add ASF license for newly created files in standalone-metastore > --- > > Key: HIVE-25774 > URL: https://issues.apache.org/jira/browse/HIVE-25774 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25774) Add ASF license for newly created files in standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-25774?focusedWorklogId=691739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691739 ] ASF GitHub Bot logged work on HIVE-25774: - Author: ASF GitHub Bot Created on: 07/Dec/21 13:18 Start Date: 07/Dec/21 13:18 Worklog Time Spent: 10m Work Description: pvary merged pull request #2844: URL: https://github.com/apache/hive/pull/2844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691739) Time Spent: 50m (was: 40m) > Add ASF license for newly created files in standalone-metastore > --- > > Key: HIVE-25774 > URL: https://issues.apache.org/jira/browse/HIVE-25774 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 4.0.0 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454445#comment-17454445
]
Viktor Csomor edited comment on HIVE-25716 at 12/7/21, 10:50 AM:
-
The root cause of the failure
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
{code}
The problem is about rounding the current time to seconds in different systems.
- Test -> System.currentTimeInMillies()
- Db metric -> NOW() / 1000
The above two instants might have differences in time.
was (Author: vcsomor):
The root cause of the failure
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
> (!) After turning off the test the problematic line is actually 215 in the
> codebase
> {code}
> Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
> >= youngDiff);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25782) Create Table As Select fails for managed ACID tables
[
https://issues.apache.org/jira/browse/HIVE-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Juhász updated HIVE-25782:
Description:
Create Table As Select fails for managed ACID tables:
*MetaException(message:Processor has no capabilities, cannot create an ACID
table.)*
HMSHandler.translate_table_dryrun invokes
MetastoreDefaultTransformer.transformCreateTable with null
processorCapabilities and processorId.
https://github.com/apache/hive/blob/c7fdd459305f4bf6913dc4bed7e8df8c7bf9e458/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L2251
{code:java}
Dec 06 05:32:47 Starting translation for CreateTable for processor null with
null on table vectortab10korc
Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
create an ACID table.)
at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Reproduction ([^ctas_acid_managed.q]):
{code:java}
set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
create table test stored as orc tblproperties ('transactional'='true') as
select from_unixtime(unix_timestamp("0002-01-01 09:57:21", "-MM-dd
HH:mm:ss")); {code}
was:
Create Table As Select fails for managed ACID tables:
*MetaException(message:Processor has no capabilities, cannot create an ACID
table.)*
{code:java}
Dec 06 05:32:47 Starting translation for CreateTable for processor null with
null on table vectortab10korc
Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
create an ACID table.)
at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
[jira] [Updated] (HIVE-25782) Create Table As Select fails for managed ACID tables
[
https://issues.apache.org/jira/browse/HIVE-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Juhász updated HIVE-25782:
Description:
Create Table As Select fails for managed ACID tables:
*MetaException(message:Processor has no capabilities, cannot create an ACID
table.)*
{code:java}
Dec 06 05:32:47 Starting translation for CreateTable for processor null with
null on table vectortab10korc
Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
create an ACID table.)
at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
Reproduction ([^ctas_acid_managed.q]):
{code:java}
set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
create table test stored as orc tblproperties ('transactional'='true') as
select from_unixtime(unix_timestamp("0002-01-01 09:57:21", "-MM-dd
HH:mm:ss")); {code}
was:
Create Table As Select fails for managed ACID tables:
*MetaException(message:Processor has no capabilities, cannot create an ACID
table.)*
{code:java}
Dec 06 05:32:47 Starting translation for CreateTable for processor null with
null on table vectortab10korc
Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
create an ACID table.) at
org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source) at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
at
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Description: In the case of accumulated entries for the same table/partition, they should be processed one by one in ASC order of write_id's. However, to support multi-threaded processing in the Cleaner, we need to make sure that the entries from the same group are not processed concurrently by multiple threads. One of the solutions that don't involve synchronization (check if already in flight, re-try when possible) would be to move the entries from the same group to the next Cleaner cycle. was: In the case of accumulated entries for the same table/partition, they should be processed one by one in ASC order of write_id's. However, to support multi-threaded processing in the Cleaner, we need to make sure that the entries from the same group are not processed concurrently by multiple threads. One of the solutions that don't involve synchronization would be to move the entries from the same group to the next Cleaner cycle. > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. > One of the solutions that don't involve synchronization (check if already in > flight, re-try when possible) would be to move the entries from the same > group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Description: In the case of accumulated entries for the same table/partition, they should be processed one by one in ASC order of write_id's. However, to support multi-threaded processing in the Cleaner, we need to make sure that the entries from the same group are not processed concurrently by multiple threads. One of the solutions that doesn't involve synchronization would be to move the entries from the same group to the next Cleaner cycle. > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. One of the solutions that doesn't involve synchronization > would be to move the entries from the same group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Description: In the case of accumulated entries for the same table/partition, they should be processed one by one in ASC order of write_id's. However, to support multi-threaded processing in the Cleaner, we need to make sure that the entries from the same group are not processed concurrently by multiple threads. One of the solutions that don't involve synchronization would be to move the entries from the same group to the next Cleaner cycle. was: In the case of accumulated entries for the same table/partition, they should be processed one by one in ASC order of write_id's. However, to support multi-threaded processing in the Cleaner, we need to make sure that the entries from the same group are not processed concurrently by multiple threads. One of the solutions that doesn't involve synchronization would be to move the entries from the same group to the next Cleaner cycle. > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. > One of the solutions that don't involve synchronization would be to move the > entries from the same group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Summary: Restore multi-threaded support in Cleaner after HIVE-25115 (was: Restore multi-threaded support in Cleaner) > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-25781: -- Parent: (was: HIVE-25115) Issue Type: Bug (was: Sub-task) > Restore multi-threaded support in Cleaner > - > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[
https://issues.apache.org/jira/browse/HIVE-25115?focusedWorklogId=691584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691584
]
ASF GitHub Bot logged work on HIVE-25115:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 09:17
Start Date: 07/Dec/21 09:17
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2825:
URL: https://github.com/apache/hive/pull/2825#discussion_r763787981
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
##
@@ -1621,6 +1621,7 @@ public void mmTableOpenWriteId() throws Exception {
verifyFooBarResult(tblName, 2);
verifyHasBase(table.getSd(), fs, "base_005_v016");
runCleaner(conf);
+runCleaner(conf);
Review comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691584)
Time Spent: 4h 20m (was: 4h 10m)
> Compaction queue entries may accumulate in "ready for cleaning" state
> -
>
> Key: HIVE-25115
> URL: https://issues.apache.org/jira/browse/HIVE-25115
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> If the Cleaner does not delete any files, the compaction queue entry is
> thrown back to the queue and remains in "ready for cleaning" state.
> Problem: If 2 compactions run on the same table and enter "ready for
> cleaning" state at the same time, only one "cleaning" will remove obsolete
> files, the other entry will remain in the queue in "ready for cleaning" state.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[
https://issues.apache.org/jira/browse/HIVE-25115?focusedWorklogId=691576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691576
]
ASF GitHub Bot logged work on HIVE-25115:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 09:04
Start Date: 07/Dec/21 09:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2825:
URL: https://github.com/apache/hive/pull/2825#discussion_r763777180
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -323,57 +323,62 @@ public void markCompacted(CompactionInfo info) throws
MetaException {
@Override
@RetrySemantics.ReadOnly
public List findReadyToClean(long minOpenTxnWaterMark, long
retentionTime) throws MetaException {
-Connection dbConn = null;
-List rc = new ArrayList<>();
-
-Statement stmt = null;
-ResultSet rs = null;
try {
- try {
-dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-stmt = dbConn.createStatement();
+ List rc = new ArrayList<>();
+
+ try (Connection dbConn =
getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+ Statement stmt = dbConn.createStatement()) {
/*
* By filtering on minOpenTxnWaterMark, we will only cleanup after
every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up
everything that was made obsolete by this compaction.
*/
-String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", "
-+ "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM
\"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
-+ READY_FOR_CLEANING + "'";
+String whereClause = " WHERE \"CQ_STATE\" = '" + READY_FOR_CLEANING +
"'";
if (minOpenTxnWaterMark > 0) {
- s = s + " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark + " OR
\"CQ_NEXT_TXN_ID\" IS NULL)";
+ whereClause += " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark
+ " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
}
if (retentionTime > 0) {
- s = s + " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct) + " -
" + retentionTime + ")";
+ whereClause += " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct)
+ " - " + retentionTime + ")";
}
-s = s + " ORDER BY \"CQ_HIGHEST_WRITE_ID\", \"CQ_ID\"";
+String s = "SELECT \"CQ_ID\", \"cq1\".\"CQ_DATABASE\",
\"cq1\".\"CQ_TABLE\", \"cq1\".\"CQ_PARTITION\"," +
+ " \"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\",
\"CQ_TBLPROPERTIES\"" +
+ " FROM \"COMPACTION_QUEUE\" \"cq1\" " +
+ "INNER JOIN (" +
+ " SELECT MIN(\"CQ_HIGHEST_WRITE_ID\") \"WRITE_ID\",
\"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\"" +
+ " FROM \"COMPACTION_QUEUE\""
+ + whereClause +
+ " GROUP BY \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\") \"cq2\"
" +
+ "ON \"cq1\".\"CQ_DATABASE\" = \"cq2\".\"CQ_DATABASE\""+
+ " AND \"cq1\".\"CQ_TABLE\" = \"cq2\".\"CQ_TABLE\""+
+ " AND (\"cq1\".\"CQ_PARTITION\" = \"cq2\".\"CQ_PARTITION\"" +
+ "OR \"cq1\".\"CQ_PARTITION\" IS NULL AND
\"cq2\".\"CQ_PARTITION\" IS NULL)"
+ + whereClause +
+ " AND \"CQ_HIGHEST_WRITE_ID\" = \"WRITE_ID\"" +
+ " ORDER BY \"CQ_ID\"";
LOG.debug("Going to execute query <" + s + ">");
-rs = stmt.executeQuery(s);
-while (rs.next()) {
- CompactionInfo info = new CompactionInfo();
- info.id = rs.getLong(1);
- info.dbname = rs.getString(2);
- info.tableName = rs.getString(3);
- info.partName = rs.getString(4);
- info.type = dbCompactionType2ThriftType(rs.getString(5).charAt(0));
- info.runAs = rs.getString(6);
- info.highestWriteId = rs.getLong(7);
- if (LOG.isDebugEnabled()) {
-LOG.debug("Found ready to clean: " + info.toString());
+try (ResultSet rs = stmt.executeQuery(s)) {
+ while (rs.next()) {
+CompactionInfo info = new CompactionInfo();
+info.id = rs.getLong(1);
+info.dbname = rs.getString(2);
+info.tableName = rs.getString(3);
+info.partName = rs.getString(4);
+info.type = dbCompactionType2ThriftType(rs.getString(5).charAt(0));
+info.runAs = rs.getString(6);
+info.highestWriteId = rs.getLong(7);
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Found ready to clean: " + info.toString());
+}
+rc.add(info);
}
- rc.add(info);
}
return rc;
} catch (SQLException e) {
LOG.error("Unable to select next element for
[jira] [Work logged] (HIVE-25115) Compaction queue entries may accumulate in "ready for cleaning" state
[
https://issues.apache.org/jira/browse/HIVE-25115?focusedWorklogId=691575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691575
]
ASF GitHub Bot logged work on HIVE-25115:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 09:04
Start Date: 07/Dec/21 09:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2825:
URL: https://github.com/apache/hive/pull/2825#discussion_r763776755
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -323,57 +323,62 @@ public void markCompacted(CompactionInfo info) throws
MetaException {
@Override
@RetrySemantics.ReadOnly
public List findReadyToClean(long minOpenTxnWaterMark, long
retentionTime) throws MetaException {
-Connection dbConn = null;
-List rc = new ArrayList<>();
-
-Statement stmt = null;
-ResultSet rs = null;
try {
- try {
-dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-stmt = dbConn.createStatement();
+ List rc = new ArrayList<>();
+
+ try (Connection dbConn =
getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+ Statement stmt = dbConn.createStatement()) {
/*
* By filtering on minOpenTxnWaterMark, we will only cleanup after
every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up
everything that was made obsolete by this compaction.
*/
-String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", "
-+ "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM
\"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
-+ READY_FOR_CLEANING + "'";
+String whereClause = " WHERE \"CQ_STATE\" = '" + READY_FOR_CLEANING +
"'";
if (minOpenTxnWaterMark > 0) {
- s = s + " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark + " OR
\"CQ_NEXT_TXN_ID\" IS NULL)";
+ whereClause += " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark
+ " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
}
if (retentionTime > 0) {
- s = s + " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct) + " -
" + retentionTime + ")";
+ whereClause += " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct)
+ " - " + retentionTime + ")";
}
-s = s + " ORDER BY \"CQ_HIGHEST_WRITE_ID\", \"CQ_ID\"";
+String s = "SELECT \"CQ_ID\", \"cq1\".\"CQ_DATABASE\",
\"cq1\".\"CQ_TABLE\", \"cq1\".\"CQ_PARTITION\"," +
+ " \"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\",
\"CQ_TBLPROPERTIES\"" +
+ " FROM \"COMPACTION_QUEUE\" \"cq1\" " +
+ "INNER JOIN (" +
Review comment:
because WRITE_ID is not unique, we can have same write_id allocated for
dif combinations of tables where it's not the latest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691575)
Time Spent: 4h (was: 3h 50m)
> Compaction queue entries may accumulate in "ready for cleaning" state
> -
>
> Key: HIVE-25115
> URL: https://issues.apache.org/jira/browse/HIVE-25115
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> If the Cleaner does not delete any files, the compaction queue entry is
> thrown back to the queue and remains in "ready for cleaning" state.
> Problem: If 2 compactions run on the same table and enter "ready for
> cleaning" state at the same time, only one "cleaning" will remove obsolete
> files, the other entry will remain in the queue in "ready for cleaning" state.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?focusedWorklogId=691567=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691567
]
ASF GitHub Bot logged work on HIVE-25716:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 08:48
Start Date: 07/Dec/21 08:48
Worklog Time Spent: 10m
Work Description: vcsomor commented on pull request #2837:
URL: https://github.com/apache/hive/pull/2837#issuecomment-987700542
Root cause was:
```java
Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
>= youngDiff);
```
Output:
```
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691567)
Time Spent: 1h (was: 50m)
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
> (!) After turning off the test the problematic line is actually 215 in the
> codebase
> {code}
> Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
> >= youngDiff);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.1
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=691553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691553 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 07/Dec/21 08:30 Start Date: 07/Dec/21 08:30 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-987685455 @ayushtkn: I've done some digging into the jline3 issue in #2617 ([here](https://github.com/apache/hive/pull/2617#issuecomment-978029623)) and I'm not sure if it was deliberate move to declare jline3 as a dependency of the `hadoop-yarn-client` artifact -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691553) Time Spent: 7.55h (was: 7h 23m) > Upgrade Hadoop to 3.3.1 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 7.55h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25733) Add check-spelling CI action
[
https://issues.apache.org/jira/browse/HIVE-25733?focusedWorklogId=691547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691547
]
ASF GitHub Bot logged work on HIVE-25733:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 08:21
Start Date: 07/Dec/21 08:21
Worklog Time Spent: 10m
Work Description: jsoref commented on pull request #2809:
URL: https://github.com/apache/hive/pull/2809#issuecomment-987679521
Yes, workflows introduced by a PR don't run in the destination repository.
You can see that it ran here:
https://github.com/jsoref/hive/commit/921dfe1b0cd4d2bdde8bdf8beea41cd37a671e95)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691547)
Time Spent: 20m (was: 10m)
> Add check-spelling CI action
>
>
> Key: HIVE-25733
> URL: https://issues.apache.org/jira/browse/HIVE-25733
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Reporter: Josh Soref
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Add CI to catch spelling errors. See [https://www.check-spelling.dev/] for
> information.
> Initially this will only check the {{serde}} directory, but the intention is
> to expand its coverage as spelling errors in other directories are fixed.
> Note that for this to work the action should be made a required check,
> otherwise when a typo is added forks from that commit will get complaints.
> If a typo is intentional, the action will provide information about how to
> add it to {{expect.txt}} such that it will be accepted as an expected item
> (i.e. not a typo).
> To skip a file/directory entirely, add a matching entry to
> {{{}excludes.txt{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24390) Spelling fixes
[ https://issues.apache.org/jira/browse/HIVE-24390?focusedWorklogId=691539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691539 ] ASF GitHub Bot logged work on HIVE-24390: - Author: ASF GitHub Bot Created on: 07/Dec/21 08:18 Start Date: 07/Dec/21 08:18 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on a change in pull request #2810: URL: https://github.com/apache/hive/pull/2810#discussion_r763734518 ## File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ## @@ -5106,25 +5106,25 @@ private static void populateLlapDaemonVarsSet(Set llapDaemonVarsSetLocal "Specifies the minimum amount of tasks, executed by a particular LLAP daemon, before the health\n" + "status of the node is examined."), LLAP_NODEHEALTHCHECKS_MININTERVALDURATION( - "hive.llap.nodehealthckecks.minintervalduration", "300s", + "hive.llap.nodehealthchecks.minintervalduration", "300s", Review comment: actually; this is a breaking change - but I don't think anyone have *ever* changed these configsso lets keep them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691539) Time Spent: 5h (was: 4h 50m) > Spelling fixes > -- > > Key: HIVE-24390 > URL: https://issues.apache.org/jira/browse/HIVE-24390 > Project: Hive > Issue Type: Bug >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viktor Csomor updated HIVE-25716:
-
Description:
Flaky check failed on run #59:
[http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
{code}
(!) After turning off the test the problematic line is actually 215 in the
codebase
{code}
Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
>= youngDiff);
{code}
was:
Flaky check failed on run #59:
[http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
{code}
(!) The problematic line is actually 215 in the codabase
{code}
Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
>= youngDiff);
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
> (!) After turning off the test the problematic line is actually 215 in the
> codebase
> {code}
> Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
> >= youngDiff);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viktor Csomor updated HIVE-25716:
-
Description:
Flaky check failed on run #59:
[http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
{code}
(!) The problematic line is actually 215 in the codabase
{code}
Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
>= youngDiff);
{code}
was:
Flaky check failed on run #59:
[http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
{code:java}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
> (!) The problematic line is actually 215 in the codabase
> {code}
> Assert.assertTrue(Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue()
> >= youngDiff);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24390) Spelling fixes
[ https://issues.apache.org/jira/browse/HIVE-24390?focusedWorklogId=691533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691533 ] ASF GitHub Bot logged work on HIVE-24390: - Author: ASF GitHub Bot Created on: 07/Dec/21 08:11 Start Date: 07/Dec/21 08:11 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on pull request #2810: URL: https://github.com/apache/hive/pull/2810#issuecomment-987671319 it seems like there were some unrelated failures? I've restarted the tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 691533) Time Spent: 4h 50m (was: 4h 40m) > Spelling fixes > -- > > Key: HIVE-24390 > URL: https://issues.apache.org/jira/browse/HIVE-24390 > Project: Hive > Issue Type: Bug >Reporter: Josh Soref >Assignee: Josh Soref >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25733) Add check-spelling CI action
[
https://issues.apache.org/jira/browse/HIVE-25733?focusedWorklogId=691531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-691531
]
ASF GitHub Bot logged work on HIVE-25733:
-
Author: ASF GitHub Bot
Created on: 07/Dec/21 08:10
Start Date: 07/Dec/21 08:10
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #2809:
URL: https://github.com/apache/hive/pull/2809#issuecomment-987670683
I don't see this action executed by github - do we need the merge this
first? iirc github action changes should run on PRs as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 691531)
Remaining Estimate: 0h
Time Spent: 10m
> Add check-spelling CI action
>
>
> Key: HIVE-25733
> URL: https://issues.apache.org/jira/browse/HIVE-25733
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Reporter: Josh Soref
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add CI to catch spelling errors. See [https://www.check-spelling.dev/] for
> information.
> Initially this will only check the {{serde}} directory, but the intention is
> to expand its coverage as spelling errors in other directories are fixed.
> Note that for this to work the action should be made a required check,
> otherwise when a typo is added forks from that commit will get complaints.
> If a typo is intentional, the action will provide information about how to
> add it to {{expect.txt}} such that it will be accepted as an expected item
> (i.e. not a typo).
> To skip a file/directory entirely, add a matching entry to
> {{{}excludes.txt{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25733) Add check-spelling CI action
[
https://issues.apache.org/jira/browse/HIVE-25733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25733:
--
Labels: pull-request-available (was: )
> Add check-spelling CI action
>
>
> Key: HIVE-25733
> URL: https://issues.apache.org/jira/browse/HIVE-25733
> Project: Hive
> Issue Type: Improvement
> Components: Build Infrastructure
>Reporter: Josh Soref
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Add CI to catch spelling errors. See [https://www.check-spelling.dev/] for
> information.
> Initially this will only check the {{serde}} directory, but the intention is
> to expand its coverage as spelling errors in other directories are fixed.
> Note that for this to work the action should be made a required check,
> otherwise when a typo is added forks from that commit will get complaints.
> If a typo is intentional, the action will provide information about how to
> add it to {{expect.txt}} such that it will be accepted as an expected item
> (i.e. not a typo).
> To skip a file/directory entirely, add a matching entry to
> {{{}excludes.txt{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454445#comment-17454445
]
Viktor Csomor edited comment on HIVE-25716 at 12/7/21, 8:03 AM:
The root cause of the failure
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
{code}
was (Author: vcsomor):
The below modified code failed locally, however it failed in the line 215.
Still investigating.
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454445#comment-17454445
]
Viktor Csomor edited comment on HIVE-25716 at 12/7/21, 8:00 AM:
The below modified code failed locally, however it failed in the line 215.
Still investigating.
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18 (which is calculated and rounded inside the runAcidMetricsService)
{code}
was (Author: vcsomor):
The below modified code failed locally, however it failed in the line 215.
Still investigating.
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (HIVE-25716) Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
[
https://issues.apache.org/jira/browse/HIVE-25716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454445#comment-17454445
]
Viktor Csomor edited comment on HIVE-25716 at 12/7/21, 7:59 AM:
The below modified code failed locally, however it failed in the line 215.
Still investigating.
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37 (diff 37578)
Young diff: 19 (19003)
Age: 18
{code}
was (Author: vcsomor):
The below modified code failed locally, however it failed in the line 215.
Still investigating.
Check:
{code}
runAcidMetricService();
long now = System.currentTimeMillis();
long oldDiff = (now - oldStart)/1000;
long youngDiff = (now - youngStart)/1000;
long threshold = 1000;
int intValue =
Metrics.getOrCreateGauge(MetricsConstants.OLDEST_READY_FOR_CLEANING_AGE).intValue();
System.err.println("Now: " + now);
System.err.println("Old start: " + oldStart);
System.err.println("Young start: " + youngStart);
System.err.println("Old diff: " + oldDiff);
System.err.println("Young diff: " + youngDiff);
System.err.println("Age: " + intValue);
boolean assertValue = (intValue <= (oldDiff + threshold));
Assert.assertTrue(assertValue);
{code}
Output
{code}
Now: 1638863282765
Old start: 1638863245187
Young start: 1638863263762
Old diff: 37
Young diff: 19
Age: 18
{code}
> Fix flaky test TestCompactionMetrics#testOldestReadyForCleaningAge
> --
>
> Key: HIVE-25716
> URL: https://issues.apache.org/jira/browse/HIVE-25716
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Karen Coppage
>Assignee: Viktor Csomor
>Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Flaky check failed on run #59:
> [http://ci.hive.apache.org/job/hive-flaky-check/467/|http://ci.hive.apache.org/job/hive-flaky-check/467/]
> {code:java}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCompactionMetrics.testOldestReadyForCleaningAge(TestCompactionMetrics.java:214)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
