[jira] [Work logged] (HIVE-24597) Replication with timestamp type partition failing in HA case with same NS
[ https://issues.apache.org/jira/browse/HIVE-24597?focusedWorklogId=532306&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532306 ] ASF GitHub Bot logged work on HIVE-24597: - Author: ASF GitHub Bot Created on: 07/Jan/21 07:54 Start Date: 07/Jan/21 07:54 Worklog Time Spent: 10m Work Description: ArkoSharma opened a new pull request #1838: URL: https://github.com/apache/hive/pull/1838 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532306) Remaining Estimate: 0h Time Spent: 10m > Replication with timestamp type partition failing in HA case with same NS > - > > Key: HIVE-24597 > URL: https://issues.apache.org/jira/browse/HIVE-24597 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24597) Replication with timestamp type partition failing in HA case with same NS
[ https://issues.apache.org/jira/browse/HIVE-24597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24597: -- Labels: pull-request-available (was: ) > Replication with timestamp type partition failing in HA case with same NS > - > > Key: HIVE-24597 > URL: https://issues.apache.org/jira/browse/HIVE-24597 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24597) Replication with timestamp type partition failing in HA case with same NS
[ https://issues.apache.org/jira/browse/HIVE-24597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arko Sharma reassigned HIVE-24597: -- > Replication with timestamp type partition failing in HA case with same NS > - > > Key: HIVE-24597 > URL: https://issues.apache.org/jira/browse/HIVE-24597 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24559) Fix some spelling issues
[ https://issues.apache.org/jira/browse/HIVE-24559?focusedWorklogId=532300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532300 ] ASF GitHub Bot logged work on HIVE-24559: - Author: ASF GitHub Bot Created on: 07/Jan/21 07:34 Start Date: 07/Jan/21 07:34 Worklog Time Spent: 10m Work Description: rickyma commented on pull request #1818: URL: https://github.com/apache/hive/pull/1818#issuecomment-755941593 Hey @vihangk1, could you please help me merge this? The patch does not have any code changes, just fixing some typo issues~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532300) Time Spent: 1.5h (was: 1h 20m) > Fix some spelling issues > > > Key: HIVE-24559 > URL: https://issues.apache.org/jira/browse/HIVE-24559 > Project: Hive > Issue Type: Improvement >Reporter: RickyMa >Priority: Trivial > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > There are some minor typos: > [https://github.com/apache/hive/pull/1805/fileshttps://github.com/apache/hive/pull/1805/fileshttps://github.com/apache/hive/blob/branch-2.3/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L858|https://github.com/apache/hive/blob/branch-2.3/metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L858] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?focusedWorklogId=532296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532296 ] ASF GitHub Bot logged work on HIVE-24550: - Author: ASF GitHub Bot Created on: 07/Jan/21 07:28 Start Date: 07/Jan/21 07:28 Worklog Time Spent: 10m Work Description: pvargacl commented on pull request #1836: URL: https://github.com/apache/hive/pull/1836#issuecomment-755939227 @pvary could you give this a second try? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532296) Time Spent: 1h 10m (was: 1h) > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24415) HiveSplitGenerator blocks Tez dispatcher
[ https://issues.apache.org/jira/browse/HIVE-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260275#comment-17260275 ] Mustafa İman commented on HIVE-24415: - Merged to master. Thank you [~rameshkumar] for review. > HiveSplitGenerator blocks Tez dispatcher > > > Key: HIVE-24415 > URL: https://issues.apache.org/jira/browse/HIVE-24415 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HiveSplitGenerator does a lot of heavyweight operations in its constructor. > These operations block AsyncDispatcher in Tez > [https://github.com/apache/tez/blob/989d286d09cac7c4e4c5a0e06dd75ea5a6f15478/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L141] > . We should move bulk of initialization out of constructor. > The only reason of setting up everything in constructor is > DynamicPartitionPruner. We can buffer incoming events in HiveSplitGenerator > until dynamic partition pruner is initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24415) HiveSplitGenerator blocks Tez dispatcher
[ https://issues.apache.org/jira/browse/HIVE-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa İman resolved HIVE-24415. - Fix Version/s: 4.0.0 Resolution: Fixed > HiveSplitGenerator blocks Tez dispatcher > > > Key: HIVE-24415 > URL: https://issues.apache.org/jira/browse/HIVE-24415 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > HiveSplitGenerator does a lot of heavyweight operations in its constructor. > These operations block AsyncDispatcher in Tez > [https://github.com/apache/tez/blob/989d286d09cac7c4e4c5a0e06dd75ea5a6f15478/tez-dag/src/main/java/org/apache/tez/dag/app/dag/RootInputInitializerManager.java#L141] > . We should move bulk of initialization out of constructor. > The only reason of setting up everything in constructor is > DynamicPartitionPruner. We can buffer incoming events in HiveSplitGenerator > until dynamic partition pruner is initialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24510) Vectorize compute_bit_vector
[ https://issues.apache.org/jira/browse/HIVE-24510?focusedWorklogId=532293&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532293 ] ASF GitHub Bot logged work on HIVE-24510: - Author: ASF GitHub Bot Created on: 07/Jan/21 06:59 Start Date: 07/Jan/21 06:59 Worklog Time Spent: 10m Work Description: abstractdog commented on a change in pull request #1824: URL: https://github.com/apache/hive/pull/1824#discussion_r553142400 ## File path: ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out ## @@ -194,7 +222,7 @@ STAGE PLANS: keys: 0 _col0 (type: int) 1 _col0 (type: int) - Statistics: Num rows: 221 Data size: 1768 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 220 Data size: 1760 Basic stats: COMPLETE Column stats: COMPLETE Review comment: have you double-checked the root cause of this one? I cannot decide at first sight whether this is a problem or not This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532293) Time Spent: 40m (was: 0.5h) > Vectorize compute_bit_vector > > > Key: HIVE-24510 > URL: https://issues.apache.org/jira/browse/HIVE-24510 > Project: Hive > Issue Type: Improvement >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > After https://issues.apache.org/jira/browse/HIVE-23530 , almost all compute > stats functions are vectorizable. Only function that is not vectorizable is > "compute_bit_vector" for ndv statistics computation. This causes "create > table as select" and "insert overwrite select" queries to run in > non-vectorized mode. > Even a very naive implementation of vectorized compute_bit_vector gives about > 50% performance improvement on simple "insert overwrite select" queries. That > is because entire mapper or reducer can run in vectorized mode. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24278) Implement an UDF for throwing exception in arbitrary vertex
[
https://issues.apache.org/jira/browse/HIVE-24278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260263#comment-17260263
]
László Bodor commented on HIVE-24278:
-
hey [~ashish-kumar-sharma]! sorry, I forgot to respond, in the mean I started
the work on [PR|https://github.com/apache/hive/pull/1817], feel free to take a
look and comment if you're interested
> Implement an UDF for throwing exception in arbitrary vertex
> ---
>
> Key: HIVE-24278
> URL: https://issues.apache.org/jira/browse/HIVE-24278
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For testing purposes sometimes we need to make the query fail in a vertex, so
> assuming that we already know the plan, it could be something like:
> on mapper side
> {code}
> select a.col1, exception_in_vertex("Map 1") from a
> join b on b.id = a.id
> {code}
> or on reducer side
> {code}
> select a.col1, exception_in_vertex("Reducer 2") from a
> join b on b.id = a.id
> {code}
> more advanced configuration is possible, e.g we want to make Reducer 2's 50th
> task's first (0th) attempt fail with an exception:
> {code}
> exception_in_vertex("Reducer 2", 50, 0)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24596) Explain ddl for debugging
[ https://issues.apache.org/jira/browse/HIVE-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260223#comment-17260223 ] Rajesh Balamohan edited comment on HIVE-24596 at 1/7/21, 4:34 AM: -- HIVE-24596 is on similar lines, but with much broader scope. This can be moved as a subtask or linked off to it later. was (Author: rajesh.balamohan): HIVE-24596 is on similar lines, but with much broader scope. This can be moved as a related or a subtask later. > Explain ddl for debugging > - > > Key: HIVE-24596 > URL: https://issues.apache.org/jira/browse/HIVE-24596 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Harshit Gupta >Priority: Major > > For debugging query issues, basic details like table schema, statistics, > partition details, query plans are needed. > It would be good to have "explain ddl" support, which can generate these > details. This can help in recreating the schema and planner issues without > sample data. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24596) Explain ddl for debugging
[ https://issues.apache.org/jira/browse/HIVE-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshit Gupta reassigned HIVE-24596: Assignee: Harshit Gupta > Explain ddl for debugging > - > > Key: HIVE-24596 > URL: https://issues.apache.org/jira/browse/HIVE-24596 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Harshit Gupta >Priority: Major > > For debugging query issues, basic details like table schema, statistics, > partition details, query plans are needed. > It would be good to have "explain ddl" support, which can generate these > details. This can help in recreating the schema and planner issues without > sample data. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24596) Explain ddl for debugging
[ https://issues.apache.org/jira/browse/HIVE-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260223#comment-17260223 ] Rajesh Balamohan commented on HIVE-24596: - HIVE-24596 is on similar lines, but with much broader scope. This can be moved as a related or a subtask later. > Explain ddl for debugging > - > > Key: HIVE-24596 > URL: https://issues.apache.org/jira/browse/HIVE-24596 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > > For debugging query issues, basic details like table schema, statistics, > partition details, query plans are needed. > It would be good to have "explain ddl" support, which can generate these > details. This can help in recreating the schema and planner issues without > sample data. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-24519.
---
Resolution: Fixed
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260206#comment-17260206
]
Krisztian Kasa commented on HIVE-24519:
---
Pushed to master. Thanks [~jcamachorodriguez], [~mgergely] for review.
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=532253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532253
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 07/Jan/21 03:57
Start Date: 07/Jan/21 03:57
Worklog Time Spent: 10m
Work Description: kasakrisz merged pull request #1772:
URL: https://github.com/apache/hive/pull/1772
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532253)
Time Spent: 2.5h (was: 2h 20m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24339) REPL LOAD command ignores config properties set by WITH clause
[
https://issues.apache.org/jira/browse/HIVE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260197#comment-17260197
]
Manish Maheshwari commented on HIVE-24339:
--
[~anishek] Fyi
> REPL LOAD command ignores config properties set by WITH clause
> --
>
> Key: HIVE-24339
> URL: https://issues.apache.org/jira/browse/HIVE-24339
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Priority: Major
>
> By debug messages we confirmed that REPL LOAD command ignored some config
> properties when they were provided in WITH clause, e.g.:
> {code}
> REPL LOAD bdpp01pub FROM
> 'hdfs://prdpdp01//apps/hive/repl/8237c7bd-ba26-4425-8659-3a0d32ab312c' WITH
> ('mapreduce.job.queuename'='default','hive.exec.parallel'='true','hive.exec.parallel.thread.number'='128',
> ...
> {code}
> We found that it was working on 16 threads, ignoring
> 'hive.exec.parallel.thread.number'='128'. Setting this property on session
> level worked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24570) Hive on spark tmp file should be delete when driver process finished
[
https://issues.apache.org/jira/browse/HIVE-24570?focusedWorklogId=532235&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532235
]
ASF GitHub Bot logged work on HIVE-24570:
-
Author: ASF GitHub Bot
Created on: 07/Jan/21 02:24
Start Date: 07/Jan/21 02:24
Worklog Time Spent: 10m
Work Description: fsilent commented on a change in pull request #1816:
URL: https://github.com/apache/hive/pull/1816#discussion_r553073159
##
File path:
spark-client/src/main/java/org/apache/hive/spark/client/AbstractSparkClient.java
##
@@ -260,7 +265,7 @@ public void cancel(String jobId) {
if (!properties.setReadable(false) || !properties.setReadable(true, true))
{
throw new IOException("Cannot change permissions of job properties
file.");
}
-properties.deleteOnExit();
+sparkTmpProperties = properties;
Review comment:
Test result like this:
1. hive on spark task is running
Queue | Application Priority | StartTime | FinishTime | State | FinalStatus
| Running Containers
-- | -- | -- | -- | -- | -- | --
Hive on Spark (sessionId = 5e75b839-0a66-45c6-bc98-4ea14befc267) | SPARK |
default | 0 | Thu Jan 7 09:52:31 +0800 2021 | Thu Jan 7 N/A | RUNNING
2. spark tmp file in java.io.tmpdir(default is /tmp)
root@ tmp]# ll spark*
-rw--- 1 xxx 37438 Jan 7 09:56
spark-submit.2308482618955056124.properties
3. beeline is finished
Hive on Spark (sessionId = 6228c803-55be-4c2a-9868-3deae7c432c3) | SPARK |
default | 0 | Thu Jan 7 09:52:31 +0800 2021 | Thu Jan 7 09:54:04 +0800 2021 |
FINISHED | SUCCEEDED
4. spark tmp file was deleted
[root@ tmp]# ll spark*
ls: cannot access spark*: No such file or directory
5. if use kill -9 beeline pid, also can delete spark tmp file
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532235)
Time Spent: 1h 10m (was: 1h)
> Hive on spark tmp file should be delete when driver process finished
>
>
> Key: HIVE-24570
> URL: https://issues.apache.org/jira/browse/HIVE-24570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0
>Reporter: zhaolong
>Assignee: zhaolong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-24570.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Hive on spark tmp file should be delete when driver process finished, now
> it`s in java.io.tmpdir (default /tmp) directory until hiveserver jvm is stop
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24570) Hive on spark tmp file should be delete when driver process finished
[
https://issues.apache.org/jira/browse/HIVE-24570?focusedWorklogId=532229&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532229
]
ASF GitHub Bot logged work on HIVE-24570:
-
Author: ASF GitHub Bot
Created on: 07/Jan/21 01:42
Start Date: 07/Jan/21 01:42
Worklog Time Spent: 10m
Work Description: fsilent commented on a change in pull request #1816:
URL: https://github.com/apache/hive/pull/1816#discussion_r553061897
##
File path:
spark-client/src/main/java/org/apache/hive/spark/client/AbstractSparkClient.java
##
@@ -260,7 +265,7 @@ public void cancel(String jobId) {
if (!properties.setReadable(false) || !properties.setReadable(true, true))
{
throw new IOException("Cannot change permissions of job properties
file.");
}
-properties.deleteOnExit();
+sparkTmpProperties = properties;
Review comment:
not only cancel, when spark driver stop include success finished and
killed, both will be delete this applicationId tmp file
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532229)
Time Spent: 1h (was: 50m)
> Hive on spark tmp file should be delete when driver process finished
>
>
> Key: HIVE-24570
> URL: https://issues.apache.org/jira/browse/HIVE-24570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0
>Reporter: zhaolong
>Assignee: zhaolong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-24570.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Hive on spark tmp file should be delete when driver process finished, now
> it`s in java.io.tmpdir (default /tmp) directory until hiveserver jvm is stop
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24311) Rowcontainer should reset readBlocks when we clear rows to prevent OOM.
[ https://issues.apache.org/jira/browse/HIVE-24311?focusedWorklogId=532218&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532218 ] ASF GitHub Bot logged work on HIVE-24311: - Author: ASF GitHub Bot Created on: 07/Jan/21 01:13 Start Date: 07/Jan/21 01:13 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1609: URL: https://github.com/apache/hive/pull/1609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532218) Time Spent: 40m (was: 0.5h) > Rowcontainer should reset readBlocks when we clear rows to prevent OOM. > --- > > Key: HIVE-24311 > URL: https://issues.apache.org/jira/browse/HIVE-24311 > Project: Hive > Issue Type: Bug >Affects Versions: All Versions >Reporter: Qiang.Kang >Assignee: Qiang.Kang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Hi, We found that `Rowcontainer.clearRows()` only reset some index, such as: > `addCursor`, `itrCursor`, etc, without resetting read blocks. > As we all know, `currentReadBlock` and `currentWriteBlock` is the main memory > usage of a `Rowcontainer` and might be very large, depending on the data > pattern. > `currentReadBlock` and `currentWriteBlock` won't be the same object after > rowcontainer flushed data to disk. > Resetting `currentReadBlock` and `currentWriteBlock` while clearing rows will > prevent OOM. > > Therefore, I submit a patch to reset read blocks for `Rowcontainer`, just > like `PTFRowcontainer` does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-15820) comment at the head of beeline -e
[
https://issues.apache.org/jira/browse/HIVE-15820?focusedWorklogId=532217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532217
]
ASF GitHub Bot logged work on HIVE-15820:
-
Author: ASF GitHub Bot
Created on: 07/Jan/21 01:07
Start Date: 07/Jan/21 01:07
Worklog Time Spent: 10m
Work Description: ujc714 commented on a change in pull request #1814:
URL: https://github.com/apache/hive/pull/1814#discussion_r553051915
##
File path: beeline/src/test/org/apache/hive/beeline/TestCommands.java
##
@@ -30,21 +30,35 @@
@Test
public void testLinesEndingWithComments() {
-int[] escape = {-1};
-assertEquals("show tables;", removeComments("show tables;",escape));
-assertEquals("show tables;", removeComments("show tables;
--comments",escape));
-assertEquals("show tables;", removeComments("show tables;
---comments",escape));
-assertEquals("show tables;", removeComments("show tables;
---comments;one;two;three",escape));
-assertEquals("show", removeComments("show-- tables;
---comments",escape));
-assertEquals("show", removeComments("show --tables;
---comments",escape));
-assertEquals("s", removeComments("s--how --tables;
---comments",escape));
-assertEquals("", removeComments("-- show tables; ---comments",escape));
+assertEquals("show tables;", removeComments("show tables;"));
+assertEquals("show tables;", removeComments("show tables; --comments"));
+assertEquals("show tables;", removeComments("show tables;
---comments"));
+assertEquals("show tables;", removeComments("show tables;
---comments;one;two;three"));
+assertEquals("show", removeComments("show-- tables; ---comments"));
+assertEquals("show", removeComments("show --tables; ---comments"));
+assertEquals("s", removeComments("s--how --tables; ---comments"));
+assertEquals("", removeComments("-- show tables; ---comments"));
-assertEquals("\"show tables\"", removeComments("\"show tables\"
--comments",escape));
-assertEquals("\"show --comments tables\"", removeComments("\"show
--comments tables\" --comments",escape));
-assertEquals("\"'show --comments' tables\"", removeComments("\"'show
--comments' tables\" --comments",escape));
-assertEquals("'show --comments tables'", removeComments("'show --comments
tables' --comments",escape));
-assertEquals("'\"show --comments tables\"'", removeComments("'\"show
--comments tables\"' --comments",escape));
+assertEquals("\"show tables\"", removeComments("\"show tables\"
--comments"));
+assertEquals("\"show --comments tables\"", removeComments("\"show
--comments tables\" --comments"));
+assertEquals("\"'show --comments' tables\"", removeComments("\"'show
--comments' tables\" --comments"));
+assertEquals("'show --comments tables'", removeComments("'show --comments
tables' --comments"));
+assertEquals("'\"show --comments tables\"'", removeComments("'\"show
--comments tables\"' --comments"));
+
+assertEquals("show tables;", removeComments("--comments\nshow tables;"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
--comments"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
---comments"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
---comments;one;two;three"));
+assertEquals("show", removeComments("--comments\nshow-- tables;
---comments"));
+assertEquals("show", removeComments("--comments\nshow --tables;
---comments"));
+assertEquals("s", removeComments("--comments\ns--how --tables;
---comments"));
+assertEquals("", removeComments("--comments\n-- show tables;
---comments"));
+
+assertEquals("\"show tables\"", removeComments("--comments\n\"show
tables\" --comments"));
+assertEquals("\"show --comments tables\"",
removeComments("--comments\n\"show --comments tables\" --comments"));
+assertEquals("\"'show --comments' tables\"",
removeComments("--comments\n\"'show --comments' tables\" --comments"));
+assertEquals("'show --comments tables'", removeComments("--comments\n'show
--comments tables' --comments"));
+assertEquals("'\"show --comments tables\"'",
removeComments("--comments\n'\"show --comments tables\"' --comments"));
Review comment:
Thanks for having a look! I added your example. It contains both
separate comment lines and inline comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532217)
[jira] [Updated] (HIVE-24594) results_cache_invalidation2.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-24594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24594:
--
Labels: pull-request-available (was: )
> results_cache_invalidation2.q is flaky
> --
>
> Key: HIVE-24594
> URL: https://issues.apache.org/jira/browse/HIVE-24594
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> results_cache_invalidation2.q failed for me couple of times on a unrelated
> PR. Here is the error log.
> {noformat}
> ---
> Test set: org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> ---
> Tests run: 90, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 450.54 s <<<
> FAILURE! - in org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
> Time elapsed: 15.087 s <<< FAILURE!
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing results_cache_invalidation2.q ^M
> 266a267
> > A masked pattern was here
> 271a273
> > A masked pattern was here
> 273c275,276
> < Stage-0 is a root stage
> ---
> > Stage-1 is a root stage
> > Stage-0 depends on stages: Stage-1
> 275a279,365
> > Stage: Stage-1
> > Tez
> > A masked pattern was here
> > Edges:
> > Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> > Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
> > A masked pattern was here
> > Vertices:
> > Map 1
> > Map Operator Tree:
> > TableScan
> > alias: tab1
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Filter Operator
> > predicate: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Select Operator
> > expressions: key (type: string)
> > outputColumnNames: _col0
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Reduce Output Operator
> > key expressions: _col0 (type: string)
> > null sort order: z
> > sort order: +
> > Map-reduce partition columns: _col0 (type: string)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Execution mode: vectorized, llap
> > LLAP IO: all inputs
> > Map 4
> > Map Operator Tree:
> > TableScan
> > alias: tab2
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 500 Data size: 43500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Fil^M
> {noformat}
> The test works for me locally. In fact the same PR had a successful run of
> this test in a previous commit. I think we should disable this and re-enable
> it after fixing the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24594) results_cache_invalidation2.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-24594?focusedWorklogId=532118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532118
]
ASF GitHub Bot logged work on HIVE-24594:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 20:30
Start Date: 06/Jan/21 20:30
Worklog Time Spent: 10m
Work Description: vihangk1 opened a new pull request #1837:
URL: https://github.com/apache/hive/pull/1837
The PR disables results_cache_invalidation2.q which fails intermittently on
unrelated PRs.
### What changes were proposed in this pull request?
Disable TestMiniLlapLocalCliDriver[results_cache_invalidation2.q] due to
flakiness.
### Why are the changes needed?
Reduce unrelated intermittent errors on the PRs to improve dev productivity.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran the split locally and made sure that the test is not run.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532118)
Remaining Estimate: 0h
Time Spent: 10m
> results_cache_invalidation2.q is flaky
> --
>
> Key: HIVE-24594
> URL: https://issues.apache.org/jira/browse/HIVE-24594
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> results_cache_invalidation2.q failed for me couple of times on a unrelated
> PR. Here is the error log.
> {noformat}
> ---
> Test set: org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> ---
> Tests run: 90, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 450.54 s <<<
> FAILURE! - in org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
> Time elapsed: 15.087 s <<< FAILURE!
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing results_cache_invalidation2.q ^M
> 266a267
> > A masked pattern was here
> 271a273
> > A masked pattern was here
> 273c275,276
> < Stage-0 is a root stage
> ---
> > Stage-1 is a root stage
> > Stage-0 depends on stages: Stage-1
> 275a279,365
> > Stage: Stage-1
> > Tez
> > A masked pattern was here
> > Edges:
> > Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> > Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
> > A masked pattern was here
> > Vertices:
> > Map 1
> > Map Operator Tree:
> > TableScan
> > alias: tab1
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Filter Operator
> > predicate: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Select Operator
> > expressions: key (type: string)
> > outputColumnNames: _col0
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Reduce Output Operator
> > key expressions: _col0 (type: string)
> > null sort order: z
> > sort order: +
> > Map-reduce partition columns: _col0 (type: string)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Execution mode: vectorized, llap
> > LLAP IO: all inputs
> > Map 4
> > Map Operator Tree:
> > TableScan
> > alias: tab2
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 500 Data size: 43500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Fil^M
> {noformat}
> The test works for me locally. In fact the same PR had a successful run of
> this test in a previous commi
[jira] [Work logged] (HIVE-24570) Hive on spark tmp file should be delete when driver process finished
[
https://issues.apache.org/jira/browse/HIVE-24570?focusedWorklogId=532112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532112
]
ASF GitHub Bot logged work on HIVE-24570:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 20:23
Start Date: 06/Jan/21 20:23
Worklog Time Spent: 10m
Work Description: yongzhi commented on a change in pull request #1816:
URL: https://github.com/apache/hive/pull/1816#discussion_r552943505
##
File path:
spark-client/src/main/java/org/apache/hive/spark/client/AbstractSparkClient.java
##
@@ -260,7 +265,7 @@ public void cancel(String jobId) {
if (!properties.setReadable(false) || !properties.setReadable(true, true))
{
throw new IOException("Cannot change permissions of job properties
file.");
}
-properties.deleteOnExit();
+sparkTmpProperties = properties;
Review comment:
Is sparkTmpProperties only assigned when cancel called? Is there any
normal scenario the properties need to be deleted?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532112)
Time Spent: 50m (was: 40m)
> Hive on spark tmp file should be delete when driver process finished
>
>
> Key: HIVE-24570
> URL: https://issues.apache.org/jira/browse/HIVE-24570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 4.0.0
>Reporter: zhaolong
>Assignee: zhaolong
>Priority: Major
> Labels: pull-request-available
> Attachments: 0001-HIVE-24570.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Hive on spark tmp file should be delete when driver process finished, now
> it`s in java.io.tmpdir (default /tmp) directory until hiveserver jvm is stop
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24594) results_cache_invalidation2.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-24594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-24594:
--
> results_cache_invalidation2.q is flaky
> --
>
> Key: HIVE-24594
> URL: https://issues.apache.org/jira/browse/HIVE-24594
> Project: Hive
> Issue Type: Test
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
>
> results_cache_invalidation2.q failed for me couple of times on a unrelated
> PR. Here is the error log.
> {noformat}
> ---
> Test set: org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> ---
> Tests run: 90, Failures: 1, Errors: 0, Skipped: 6, Time elapsed: 450.54 s <<<
> FAILURE! - in org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver
> org.apache.hadoop.hive.cli.split19.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
> Time elapsed: 15.087 s <<< FAILURE!
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing results_cache_invalidation2.q ^M
> 266a267
> > A masked pattern was here
> 271a273
> > A masked pattern was here
> 273c275,276
> < Stage-0 is a root stage
> ---
> > Stage-1 is a root stage
> > Stage-0 depends on stages: Stage-1
> 275a279,365
> > Stage: Stage-1
> > Tez
> > A masked pattern was here
> > Edges:
> > Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> > Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
> > A masked pattern was here
> > Vertices:
> > Map 1
> > Map Operator Tree:
> > TableScan
> > alias: tab1
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Filter Operator
> > predicate: key is not null (type: boolean)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Select Operator
> > expressions: key (type: string)
> > outputColumnNames: _col0
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Reduce Output Operator
> > key expressions: _col0 (type: string)
> > null sort order: z
> > sort order: +
> > Map-reduce partition columns: _col0 (type: string)
> > Statistics: Num rows: 1500 Data size: 130500 Basic
> > stats: COMPLETE Column stats: COMPLETE
> > Execution mode: vectorized, llap
> > LLAP IO: all inputs
> > Map 4
> > Map Operator Tree:
> > TableScan
> > alias: tab2
> > filterExpr: key is not null (type: boolean)
> > Statistics: Num rows: 500 Data size: 43500 Basic stats:
> > COMPLETE Column stats: COMPLETE
> > Fil^M
> {noformat}
> The test works for me locally. In fact the same PR had a successful run of
> this test in a previous commit. I think we should disable this and re-enable
> it after fixing the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21737) Upgrade Avro to version 1.10.1
[ https://issues.apache.org/jira/browse/HIVE-21737?focusedWorklogId=532090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532090 ] ASF GitHub Bot logged work on HIVE-21737: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:33 Start Date: 06/Jan/21 19:33 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #1806: URL: https://github.com/apache/hive/pull/1806#issuecomment-755568148 Thanks @heuermh ! I'll start a vote on Hive 2.3.8 soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532090) Time Spent: 8h 20m (was: 8h 10m) > Upgrade Avro to version 1.10.1 > -- > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: > 0001-HIVE-21737-Make-Avro-use-in-Hive-compatible-with-Avr.patch > > Time Spent: 8h 20m > Remaining Estimate: 0h > > Avro >= 1.9.x bring a lot of fixes including a leaner version of Avro without > Jackson in the public API and Guava as a dependency. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21737) Upgrade Avro to version 1.10.1
[ https://issues.apache.org/jira/browse/HIVE-21737?focusedWorklogId=532086&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532086 ] ASF GitHub Bot logged work on HIVE-21737: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:26 Start Date: 06/Jan/21 19:26 Worklog Time Spent: 10m Work Description: heuermh commented on pull request #1806: URL: https://github.com/apache/hive/pull/1806#issuecomment-76893 @sunchao With the changes in bigdatagenomics/adam#2289 to remove various workarounds, the combination in pull request https://github.com/apache/spark/pull/30517 works for us. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532086) Time Spent: 8h 10m (was: 8h) > Upgrade Avro to version 1.10.1 > -- > > Key: HIVE-21737 > URL: https://issues.apache.org/jira/browse/HIVE-21737 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ismaël Mejía >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Attachments: > 0001-HIVE-21737-Make-Avro-use-in-Hive-compatible-with-Avr.patch > > Time Spent: 8h 10m > Remaining Estimate: 0h > > Avro >= 1.9.x bring a lot of fixes including a leaner version of Avro without > Jackson in the public API and Guava as a dependency. Worth the update. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24218) Drop table used by a materialized view
[
https://issues.apache.org/jira/browse/HIVE-24218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely resolved HIVE-24218.
---
Resolution: Duplicate
> Drop table used by a materialized view
> --
>
> Key: HIVE-24218
> URL: https://issues.apache.org/jira/browse/HIVE-24218
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive, HiveServer2, Metastore
>Affects Versions: 3.1.0
>Reporter: stephbat
>Priority: Critical
>
> I have discovered that it's possible to drop a table used by a materialized
> view. When I drop this table, the result is OK while I think this action
> should be refused. When I check in the metastore database, I can see that the
> table has been partially deleted (ie : the reference of the table still
> exists in TBLS and in MV_TABLES_USED). This introduces an inconsistency in
> the metastore.
> Steps to reproduced :
> {code:java}
> jdbc:hive2://localhost.> use use ptest2_db_dev;
> No rows affected (0.067 seconds)
> 0: jdbc:hive2://localhost.> create table table_blocked (id string);
> No rows affected (0.97 seconds)
> 0: jdbc:hive2://localhost.> desc table_blocked;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | id| string | |
> +---++--+
> 1 row selected (0.171 seconds)
> 0: jdbc:hive2://localhost.> create materialized view table_blocked_mv as
> select * from table_blocked;
> No rows affected (18.055 seconds)
> 0: jdbc:hive2://localhost.> desc table_blocked_mv;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | id| string | |
> +---++--+
> 1 row selected (0.316 seconds)
> 0: jdbc:hive2://localhost.> drop table table_blocked;
> No rows affected (10.803 seconds)
> 0: jdbc:hive2://localhost.> desc table_blocked_mv;
> +---++--+
> | col_name | data_type | comment |
> +---++--+
> | id| string | |
> +---++--+
> 1 row selected (0.222 seconds)
> 0: jdbc:hive2://localhost.> desc table_blocked;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> fetch table table_blocked. null (state=42000,code=4)
> 0: jdbc:hive2://localhost.> select * from table_blocked_mv;
> Error: Error while compiling statement: FAILED: SemanticException Table
> ptest2_db_dev.table_blocked not found when trying to obtain it to check
> masking/filtering policies (state=42000,code=4)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24593) Clean up checkstyle violations in ddl
[ https://issues.apache.org/jira/browse/HIVE-24593?focusedWorklogId=532078&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532078 ] ASF GitHub Bot logged work on HIVE-24593: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:09 Start Date: 06/Jan/21 19:09 Worklog Time Spent: 10m Work Description: miklosgergely opened a new pull request #1835: URL: https://github.com/apache/hive/pull/1835 ### What changes were proposed in this pull request? Clean up the checkstyle violations in the ddl package. ### Why are the changes needed? DDL is almost free of checkstlye violations, trying to keep it that way. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? All the unit tests and q tests are still running. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532078) Remaining Estimate: 0h Time Spent: 10m > Clean up checkstyle violations in ddl > - > > Key: HIVE-24593 > URL: https://issues.apache.org/jira/browse/HIVE-24593 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24593) Clean up checkstyle violations in ddl
[ https://issues.apache.org/jira/browse/HIVE-24593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24593: -- Labels: pull-request-available (was: ) > Clean up checkstyle violations in ddl > - > > Key: HIVE-24593 > URL: https://issues.apache.org/jira/browse/HIVE-24593 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?focusedWorklogId=532077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532077 ] ASF GitHub Bot logged work on HIVE-24550: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:09 Start Date: 06/Jan/21 19:09 Worklog Time Spent: 10m Work Description: pvargacl opened a new pull request #1836: URL: https://github.com/apache/hive/pull/1836 Rebased second PR of HIVE-24550 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532077) Time Spent: 1h (was: 50m) > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24593) Clean up checkstyle violations in ddl
[ https://issues.apache.org/jira/browse/HIVE-24593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-24593: - > Clean up checkstyle violations in ddl > - > > Key: HIVE-24593 > URL: https://issues.apache.org/jira/browse/HIVE-24593 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[ https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532075 ] ASF GitHub Bot logged work on HIVE-24509: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:01 Start Date: 06/Jan/21 19:01 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #1756: URL: https://github.com/apache/hive/pull/1756#issuecomment-755524799 @belugabehr removed unrelated changes from this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532075) Time Spent: 8h 10m (was: 8h) > Move show specific codes under DDL and cut MetaDataFormatter classes to pieces > -- > > Key: HIVE-24509 > URL: https://issues.apache.org/jira/browse/HIVE-24509 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 8h 10m > Remaining Estimate: 0h > > Lot of show ... specific codes are under the > org.apache.hadoop.hive.ql.metadata.formatting package which are used only by > these commands. Also the two MetaDataFormatters (JsonMetaDataFormatter, > TextMetaDataFormatter) are trying to do everything, while they contain a lot > of code duplications. Their functionalities should be put under the > directories of the appropriate show commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[
https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532073
]
ASF GitHub Bot logged work on HIVE-24509:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 19:01
Start Date: 06/Jan/21 19:01
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1756:
URL: https://github.com/apache/hive/pull/1756#discussion_r552905387
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowDbLocksAnalyzer.java
##
@@ -23,9 +23,6 @@
import org.apache.hadoop.hive.ql.ddl.DDLWork;
import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.TaskFactory;
-import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;
-import org.apache.hadoop.hive.ql.lockmgr.LockException;
-import org.apache.hadoop.hive.ql.lockmgr.TxnManagerFactory;
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AlterTableAddPartitionDesc.java
##
@@ -234,7 +233,7 @@ public void setWriteId(long writeId) {
@Override
public String getFullTableName() {
-return AcidUtils.getFullTableName(dbName,tableName);
+return AcidUtils.getFullTableName(dbName, tableName);
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/storage/skewed/AlterTableSetSkewedLocationAnalyzer.java
##
@@ -20,7 +20,6 @@
import java.net.URI;
import java.net.URISyntaxException;
-import java.util.ArrayList;
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/create/AlterViewAsAnalyzer.java
##
@@ -84,7 +84,7 @@ private void validateCreateView(AlterViewAsDesc desc,
SemanticAnalyzer analyzer)
if (oldView == null) {
String viewNotExistErrorMsg = "The following view does not exist: " +
desc.getViewName();
- throw new SemanticException(
ErrorMsg.ALTER_VIEW_AS_SELECT_NOT_EXIST.getMsg(viewNotExistErrorMsg));
+ throw new
SemanticException(ErrorMsg.ALTER_VIEW_AS_SELECT_NOT_EXIST.getMsg(viewNotExistErrorMsg));
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java
##
@@ -17,51 +17,53 @@
*/
package org.apache.hadoop.hive.ql.metadata.formatting;
-import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* Helper class to build Maps consumed by the JSON formatter. Only
* add non-null entries to the Map.
*/
-public class MapBuilder {
-private Map map = new LinkedHashMap();
+public final class MapBuilder {
+ private Map map = new LinkedHashMap();
-private MapBuilder() {}
+ private MapBuilder() {
+ }
-public static MapBuilder create() {
-return new MapBuilder();
-}
+ public static MapBuilder create() {
+return new MapBuilder();
+ }
-public MapBuilder put(String name, Object val) {
-if (val != null)
-map.put(name, val);
-return this;
+ public MapBuilder put(String name, Object val) {
+if (val != null) {
+ map.put(name, val);
}
+return this;
+ }
-public MapBuilder put(String name, boolean val) {
-map.put(name, Boolean.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, boolean val) {
+map.put(name, Boolean.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, int val) {
-map.put(name, Integer.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, int val) {
+map.put(name, Integer.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, long val) {
-map.put(name, Long.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, long val) {
+map.put(name, Long.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, T val, boolean use) {
-if (use)
-put(name, val);
-return this;
+ public MapBuilder put(String name, T val, boolean use) {
+if (use) {
+ put(name, val);
}
+return this;
+ }
-public Map build() {
-return map;
-}
+ public Map build() {
+return map;
+ }
Review comment:
Removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532073)
Time Spent: 7h 50m (was: 7h 40m)
> Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
>
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[ https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532074&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532074 ] ASF GitHub Bot logged work on HIVE-24509: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:01 Start Date: 06/Jan/21 19:01 Worklog Time Spent: 10m Work Description: miklosgergely commented on a change in pull request #1756: URL: https://github.com/apache/hive/pull/1756#discussion_r552905713 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java ## @@ -26,9 +26,6 @@ import org.apache.hadoop.hive.ql.ddl.DDLWork; import org.apache.hadoop.hive.ql.exec.Task; import org.apache.hadoop.hive.ql.exec.TaskFactory; -import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager; -import org.apache.hadoop.hive.ql.lockmgr.LockException; -import org.apache.hadoop.hive.ql.lockmgr.TxnManagerFactory; Review comment: Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532074) Time Spent: 8h (was: 7h 50m) > Move show specific codes under DDL and cut MetaDataFormatter classes to pieces > -- > > Key: HIVE-24509 > URL: https://issues.apache.org/jira/browse/HIVE-24509 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 8h > Remaining Estimate: 0h > > Lot of show ... specific codes are under the > org.apache.hadoop.hive.ql.metadata.formatting package which are used only by > these commands. Also the two MetaDataFormatters (JsonMetaDataFormatter, > TextMetaDataFormatter) are trying to do everything, while they contain a lot > of code duplications. Their functionalities should be put under the > directories of the appropriate show commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[
https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532072&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532072
]
ASF GitHub Bot logged work on HIVE-24509:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 19:00
Start Date: 06/Jan/21 19:00
Worklog Time Spent: 10m
Work Description: miklosgergely commented on a change in pull request
#1756:
URL: https://github.com/apache/hive/pull/1756#discussion_r552905134
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java
##
@@ -31,9 +31,11 @@
public class ShowCompactionsDesc implements DDLDesc, Serializable {
private static final long serialVersionUID = 1L;
+ // @formatter:off
public static final String SCHEMA =
"compactionid,dbname,tabname,partname,type,state,hostname,workerid,enqueuetime,starttime,duration,hadoopjobid,errormessage#"
+
"string:string:string:string:string:string:string:string:string:string:string:string:string";
+ // @formatter:on
Review comment:
Removed.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckAnalyzer.java
##
@@ -68,7 +68,8 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
}
Table table = getTable(tableName);
-Map> partitionSpecs =
ParseUtils.getFullPartitionSpecs(root, table, conf, false);
+Map> partitionSpecs =
ParseUtils.getFullPartitionSpecs(root, table, conf,
+false);
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/constraint/add/AlterTableAddConstraintAnalyzer.java
##
@@ -79,7 +79,8 @@ protected void analyzeCommand(TableName tableName,
Map partition
throw new
SemanticException(ErrorMsg.NOT_RECOGNIZED_CONSTRAINT.getMsg(constraintNode.getToken().getText()));
}
-Constraints constraints = new Constraints(primaryKeys, foreignKeys, null,
uniqueConstraints, null, checkConstraints);
+Constraints constraints =
+new Constraints(primaryKeys, foreignKeys, null, uniqueConstraints,
null, checkConstraints);
Review comment:
Removed.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableOperation.java
##
@@ -35,7 +35,6 @@
import org.apache.hadoop.hive.ql.ddl.DDLOperationContext;
import org.apache.hadoop.hive.ql.ddl.DDLUtils;
import
org.apache.hadoop.hive.ql.ddl.table.constraint.add.AlterTableAddConstraintOperation;
-import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
Review comment:
Removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532072)
Time Spent: 7h 40m (was: 7.5h)
> Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
> --
>
> Key: HIVE-24509
> URL: https://issues.apache.org/jira/browse/HIVE-24509
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Lot of show ... specific codes are under the
> org.apache.hadoop.hive.ql.metadata.formatting package which are used only by
> these commands. Also the two MetaDataFormatters (JsonMetaDataFormatter,
> TextMetaDataFormatter) are trying to do everything, while they contain a lot
> of code duplications. Their functionalities should be put under the
> directories of the appropriate show commands.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[ https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532070&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532070 ] ASF GitHub Bot logged work on HIVE-24509: - Author: ASF GitHub Bot Created on: 06/Jan/21 19:00 Start Date: 06/Jan/21 19:00 Worklog Time Spent: 10m Work Description: miklosgergely commented on a change in pull request #1756: URL: https://github.com/apache/hive/pull/1756#discussion_r552905074 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/drop/AbstractDropPartitionAnalyzer.java ## @@ -20,25 +20,17 @@ import java.util.ArrayList; import java.util.Collection; -import java.util.HashMap; Review comment: Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 532070) Time Spent: 7.5h (was: 7h 20m) > Move show specific codes under DDL and cut MetaDataFormatter classes to pieces > -- > > Key: HIVE-24509 > URL: https://issues.apache.org/jira/browse/HIVE-24509 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Time Spent: 7.5h > Remaining Estimate: 0h > > Lot of show ... specific codes are under the > org.apache.hadoop.hive.ql.metadata.formatting package which are used only by > these commands. Also the two MetaDataFormatters (JsonMetaDataFormatter, > TextMetaDataFormatter) are trying to do everything, while they contain a lot > of code duplications. Their functionalities should be put under the > directories of the appropriate show commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24543) Support SAML 2.0 as an authentication mechanism
[ https://issues.apache.org/jira/browse/HIVE-24543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250704#comment-17250704 ] Vihang Karajgaonkar edited comment on HIVE-24543 at 1/6/21, 6:54 PM: - I have created a design doc which can be reviewed here: https://cwiki.apache.org/confluence/display/Hive/HIVE-24543%3A+Support+SAML+2.0+authentication+mode was (Author: vihangk1): I have created a design doc which can be reviewed here: https://cwiki.apache.org/confluence/display/Hive/Support+SAML+2.0+authentication+mode > Support SAML 2.0 as an authentication mechanism > --- > > Key: HIVE-24543 > URL: https://issues.apache.org/jira/browse/HIVE-24543 > Project: Hive > Issue Type: New Feature >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > With cloud based deployments, having a SAML 2.0 based authentication support > in HS2 will be greatly useful in case of federated or external identity > providers like Okta, PingIdentity or Azure AD. > This authentication mechanism can initially be only supported on http > transport mode in HiveServer2 since the SAML 2.0 protocol is primarily > designed for web clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24509) Move show specific codes under DDL and cut MetaDataFormatter classes to pieces
[
https://issues.apache.org/jira/browse/HIVE-24509?focusedWorklogId=532056&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532056
]
ASF GitHub Bot logged work on HIVE-24509:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 18:37
Start Date: 06/Jan/21 18:37
Worklog Time Spent: 10m
Work Description: belugabehr commented on a change in pull request #1756:
URL: https://github.com/apache/hive/pull/1756#discussion_r552889107
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/drop/AbstractDropPartitionAnalyzer.java
##
@@ -20,25 +20,17 @@
import java.util.ArrayList;
import java.util.Collection;
-import java.util.HashMap;
Review comment:
All changes in this file are out of scope of the PR. Please revert.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/msck/MsckAnalyzer.java
##
@@ -68,7 +68,8 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
}
Table table = getTable(tableName);
-Map> partitionSpecs =
ParseUtils.getFullPartitionSpecs(root, table, conf, false);
+Map> partitionSpecs =
ParseUtils.getFullPartitionSpecs(root, table, conf,
+false);
Review comment:
All changes in this file are out of scope of the PR. Please revert.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableOperation.java
##
@@ -35,7 +35,6 @@
import org.apache.hadoop.hive.ql.ddl.DDLOperationContext;
import org.apache.hadoop.hive.ql.ddl.DDLUtils;
import
org.apache.hadoop.hive.ql.ddl.table.constraint.add.AlterTableAddConstraintOperation;
-import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
Review comment:
All changes in this file are out of scope of the PR. Please revert.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/lock/show/ShowLocksAnalyzer.java
##
@@ -26,9 +26,6 @@
import org.apache.hadoop.hive.ql.ddl.DDLWork;
import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.TaskFactory;
-import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;
-import org.apache.hadoop.hive.ql.lockmgr.LockException;
-import org.apache.hadoop.hive.ql.lockmgr.TxnManagerFactory;
Review comment:
All changes in this file are out of scope of the PR. Please revert.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MapBuilder.java
##
@@ -17,51 +17,53 @@
*/
package org.apache.hadoop.hive.ql.metadata.formatting;
-import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* Helper class to build Maps consumed by the JSON formatter. Only
* add non-null entries to the Map.
*/
-public class MapBuilder {
-private Map map = new LinkedHashMap();
+public final class MapBuilder {
+ private Map map = new LinkedHashMap();
-private MapBuilder() {}
+ private MapBuilder() {
+ }
-public static MapBuilder create() {
-return new MapBuilder();
-}
+ public static MapBuilder create() {
+return new MapBuilder();
+ }
-public MapBuilder put(String name, Object val) {
-if (val != null)
-map.put(name, val);
-return this;
+ public MapBuilder put(String name, Object val) {
+if (val != null) {
+ map.put(name, val);
}
+return this;
+ }
-public MapBuilder put(String name, boolean val) {
-map.put(name, Boolean.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, boolean val) {
+map.put(name, Boolean.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, int val) {
-map.put(name, Integer.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, int val) {
+map.put(name, Integer.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, long val) {
-map.put(name, Long.valueOf(val));
-return this;
-}
+ public MapBuilder put(String name, long val) {
+map.put(name, Long.valueOf(val));
+return this;
+ }
-public MapBuilder put(String name, T val, boolean use) {
-if (use)
-put(name, val);
-return this;
+ public MapBuilder put(String name, T val, boolean use) {
+if (use) {
+ put(name, val);
}
+return this;
+ }
-public Map build() {
-return map;
-}
+ public Map build() {
+return map;
+ }
Review comment:
All changes (there are several) in this file are out of scope of the PR.
Please revert.
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/add/AlterTableAddPartitionDesc.java
##
@@ -234,7 +233,7 @@ public void setWriteId(long writeId) {
@Override
public String getFullTableName() {
-return AcidUtils.getFullTableName(dbName,tableName);
+return AcidUt
[jira] [Updated] (HIVE-24583) Make Hive JDBC Driver Java Executable
[
https://issues.apache.org/jira/browse/HIVE-24583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-24583:
--
Priority: Minor (was: Major)
> Make Hive JDBC Driver Java Executable
> -
>
> Key: HIVE-24583
> URL: https://issues.apache.org/jira/browse/HIVE-24583
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: wenjun ma
>Priority: Minor
>
> Runining:
> {code:none}
> java -jar hive-jdbc.jar
> {code}
> Should print driver version information. Something like this is implemented,
> but probably better to move it into a {{main}} method in {{HiveDriver}} class.
> https://github.com/apache/hive/blob/72d983ae76f420bdb719d33002a9c321a4e4f891/jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java#L1218-L1222
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24583) Make Hive JDBC Driver Java Executable
[
https://issues.apache.org/jira/browse/HIVE-24583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-24583:
--
Summary: Make Hive JDBC Driver Java Executable (was: Make Hive JDBC Java
Executable)
> Make Hive JDBC Driver Java Executable
> -
>
> Key: HIVE-24583
> URL: https://issues.apache.org/jira/browse/HIVE-24583
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: wenjun ma
>Priority: Major
>
> Runining:
> {code:none}
> java -jar hive-jdbc.jar
> {code}
> Should print driver version information. Something like this is implemented,
> but probably better to move it into a {{main}} method in {{HiveDriver}} class.
> https://github.com/apache/hive/blob/72d983ae76f420bdb719d33002a9c321a4e4f891/jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java#L1218-L1222
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24515) Analyze table job can be skipped when stats populated are already accurate
[
https://issues.apache.org/jira/browse/HIVE-24515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24515:
--
Labels: pull-request-available (was: )
> Analyze table job can be skipped when stats populated are already accurate
> --
>
> Key: HIVE-24515
> URL: https://issues.apache.org/jira/browse/HIVE-24515
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For non-partitioned tables, stats detail should be present in table level,
> e.g
> {noformat}
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"d_current_day":"true"...
> }}
> {noformat}
> For partitioned tables, stats detail should be present in partition level,
> {noformat}
> store_sales(ss_sold_date_sk=2451819)
> {totalSize=0, numRows=0, rawDataSize=0,
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"ss_addr_sk":"true"}}
>
> {noformat}
> When stats populated are already accurate, {{analyze table tn compute
> statistics for columns}} should skip launching the job.
>
> For ACID tables, stats are auto computed and it can skip computing stats
> again when stats are accurate.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24515) Analyze table job can be skipped when stats populated are already accurate
[
https://issues.apache.org/jira/browse/HIVE-24515?focusedWorklogId=532000&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-532000
]
ASF GitHub Bot logged work on HIVE-24515:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 17:28
Start Date: 06/Jan/21 17:28
Worklog Time Spent: 10m
Work Description: maheshk114 opened a new pull request #1834:
URL: https://github.com/apache/hive/pull/1834
…
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 532000)
Remaining Estimate: 0h
Time Spent: 10m
> Analyze table job can be skipped when stats populated are already accurate
> --
>
> Key: HIVE-24515
> URL: https://issues.apache.org/jira/browse/HIVE-24515
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For non-partitioned tables, stats detail should be present in table level,
> e.g
> {noformat}
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"d_current_day":"true"...
> }}
> {noformat}
> For partitioned tables, stats detail should be present in partition level,
> {noformat}
> store_sales(ss_sold_date_sk=2451819)
> {totalSize=0, numRows=0, rawDataSize=0,
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"ss_addr_sk":"true"}}
>
> {noformat}
> When stats populated are already accurate, {{analyze table tn compute
> statistics for columns}} should skip launching the job.
>
> For ACID tables, stats are auto computed and it can skip computing stats
> again when stats are accurate.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259883#comment-17259883
]
Stamatis Zampetakis commented on HIVE-24590:
I think a good solution is to use {{IdlePurgePolicy}} and let log4j take care
of the rest but if you have other ideas please share them.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259881#comment-17259881
]
Eugene Chung edited comment on HIVE-24590 at 1/6/21, 5:00 PM:
--
{noformat}
2021-01-06 22:35:48,150 edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd
HiveServer2-HttpHandler-Pool: Thread-102 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
for RandomAccessFileManager is stopped.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
<--- The query is done. There's a sessionId between time and the thread name.
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
<--- The session is closing. There's NO sessionId between time and the thread
name. It means log4j MDC is cleared.{noformat}
Is it safe not to create HushableRandomAccessFileAppender if log4j MDC has no
operation context?
I'm thinking of it as a solution to the leak. (+ I know where MDC contexts are
set.
https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/common/src/java/org/apache/hadoop/hive/common/LogUtils.java#L218)
was (Author: euigeun_chung):
{noformat}
2021-01-06 22:35:48,150 edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd
HiveServer2-HttpHandler-Pool: Thread-102 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
for RandomAccessFileManager is stopped.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
<--- The query is done. There's a sessionId between time and the thread name.
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
<--- The session is closing. There's NO sessionId between time and the thread
name. It means log4j MDC is cleared.{noformat}
Is it safe not to create HushableRandomAccessFileAppender if log4j MDC has no
operation context?
I'm thinking of it as a solution to the leak.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259881#comment-17259881
]
Eugene Chung commented on HIVE-24590:
-
{noformat}
2021-01-06 22:35:48,150 edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd
HiveServer2-HttpHandler-Pool: Thread-102 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
for RandomAccessFileManager is stopped.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
<--- The query is done. There's a sessionId between time and the thread name.
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
<--- The session is closing. There's NO sessionId between time and the thread
name. It means log4j MDC is cleared.{noformat}
Is it safe not to create HushableRandomAccessFileAppender if log4j MDC has no
operation context?
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259881#comment-17259881
]
Eugene Chung edited comment on HIVE-24590 at 1/6/21, 4:58 PM:
--
{noformat}
2021-01-06 22:35:48,150 edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd
HiveServer2-HttpHandler-Pool: Thread-102 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
for RandomAccessFileManager is stopped.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
<--- The query is done. There's a sessionId between time and the thread name.
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
<--- The session is closing. There's NO sessionId between time and the thread
name. It means log4j MDC is cleared.{noformat}
Is it safe not to create HushableRandomAccessFileAppender if log4j MDC has no
operation context?
I'm thinking of it as a solution to the leak.
was (Author: euigeun_chung):
{noformat}
2021-01-06 22:35:48,150 edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd
HiveServer2-HttpHandler-Pool: Thread-102 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
for RandomAccessFileManager is stopped.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223512_515ce970-bde2-4066-bebf-f97c039ece3c
<--- The query is done. There's a sessionId between time and the thread name.
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/**/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/**_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
<--- The session is closing. There's NO sessionId between time and the thread
name. It means log4j MDC is cleared.{noformat}
Is it safe not to create HushableRandomAccessFileAppender if log4j MDC has no
operation context?
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259880#comment-17259880
]
Stamatis Zampetakis commented on HIVE-24590:
bq. Then it seems that operation log file with the different name is created by
the same thread.
Given that they have a different name I suppose it means that somebody changed
the log4j context on this thread (to reflect another session/query) but
possibly at a bad timing.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24586) Rename compaction "attempted" status
[
https://issues.apache.org/jira/browse/HIVE-24586?focusedWorklogId=531970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531970
]
ASF GitHub Bot logged work on HIVE-24586:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 16:47
Start Date: 06/Jan/21 16:47
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #1831:
URL: https://github.com/apache/hive/pull/1831#discussion_r552797632
##
File path:
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java
##
@@ -1159,6 +1163,42 @@ public void resolveUserToRunAs() throws Exception {
Assert.assertNotEquals(userFromConf,
initiator.resolveUserToRunAs(tblNameOwners, t, null));
}
+ @Test public void testInitiatorFailure() throws Exception {
+String tableName = "my_table";
+Table t = newTable("default", tableName, false);
+
+HiveConf.setIntVar(conf,
HiveConf.ConfVars.HIVE_COMPACTOR_ABORTEDTXN_THRESHOLD, 1);
+
+// 2 aborts
+for (int i = 0; i < 2; i++) {
+ long txnid = openTxn();
+ LockComponent comp = new LockComponent(LockType.SHARED_WRITE,
LockLevel.TABLE, "default");
+ comp.setTablename(tableName);
+ comp.setOperationType(DataOperationType.UPDATE);
+ List components = new ArrayList(1);
+ components.add(comp);
+ LockRequest req = new LockRequest(components, "me", "localhost");
+ req.setTxnid(txnid);
+ LockResponse res = txnHandler.lock(req);
+ txnHandler.abortTxn(new AbortTxnRequest(txnid));
+}
+
+// run and fail initiator
+Initiator initiator = Mockito.spy(new Initiator());
+initiator.setThreadId((int) t.getId());
+initiator.setConf(conf);
+initiator.init(new AtomicBoolean(true));
+doThrow(new
RuntimeException()).when(initiator).resolveValidWriteIds(any());
+initiator.run();
+
+// verify status of table compaction
+ShowCompactResponse rsp = txnHandler.showCompact(new ShowCompactRequest());
+List compacts = rsp.getCompacts();
+Assert.assertEquals(1, compacts.size());
+Assert.assertEquals("did not initiate", compacts.get(0).getState());
Review comment:
Nice catch, will do!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531970)
Time Spent: 0.5h (was: 20m)
> Rename compaction "attempted" status
>
>
> Key: HIVE-24586
> URL: https://issues.apache.org/jira/browse/HIVE-24586
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> A compaction with "attempted" status sounds like compactor tried to compact
> the table/partition and failed. In reality it means one of these:
> * the Initiator did not queue compaction because the number of previously
> failed compactions has passed a threshold
> * the Initiator did not queue compaction because of an error
> In both these cases the user is still able initiate compaction manually. This
> should be made clearer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga reopened HIVE-24550: > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24586) Rename compaction "attempted" status
[
https://issues.apache.org/jira/browse/HIVE-24586?focusedWorklogId=531963&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531963
]
ASF GitHub Bot logged work on HIVE-24586:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 16:38
Start Date: 06/Jan/21 16:38
Worklog Time Spent: 10m
Work Description: pvargacl commented on a change in pull request #1831:
URL: https://github.com/apache/hive/pull/1831#discussion_r552788738
##
File path:
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java
##
@@ -1159,6 +1163,42 @@ public void resolveUserToRunAs() throws Exception {
Assert.assertNotEquals(userFromConf,
initiator.resolveUserToRunAs(tblNameOwners, t, null));
}
+ @Test public void testInitiatorFailure() throws Exception {
+String tableName = "my_table";
+Table t = newTable("default", tableName, false);
+
+HiveConf.setIntVar(conf,
HiveConf.ConfVars.HIVE_COMPACTOR_ABORTEDTXN_THRESHOLD, 1);
+
+// 2 aborts
+for (int i = 0; i < 2; i++) {
+ long txnid = openTxn();
+ LockComponent comp = new LockComponent(LockType.SHARED_WRITE,
LockLevel.TABLE, "default");
+ comp.setTablename(tableName);
+ comp.setOperationType(DataOperationType.UPDATE);
+ List components = new ArrayList(1);
+ components.add(comp);
+ LockRequest req = new LockRequest(components, "me", "localhost");
+ req.setTxnid(txnid);
+ LockResponse res = txnHandler.lock(req);
+ txnHandler.abortTxn(new AbortTxnRequest(txnid));
+}
+
+// run and fail initiator
+Initiator initiator = Mockito.spy(new Initiator());
+initiator.setThreadId((int) t.getId());
+initiator.setConf(conf);
+initiator.init(new AtomicBoolean(true));
+doThrow(new
RuntimeException()).when(initiator).resolveValidWriteIds(any());
+initiator.run();
+
+// verify status of table compaction
+ShowCompactResponse rsp = txnHandler.showCompact(new ShowCompactRequest());
+List compacts = rsp.getCompacts();
+Assert.assertEquals(1, compacts.size());
+Assert.assertEquals("did not initiate", compacts.get(0).getState());
Review comment:
I just noticed that in Initiator.scheduleCompactionIfRequired the
exception is not passed to CompactionInfo, so it won't get saved. Could you
also fix that, and here you could check if it was saved correctly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531963)
Time Spent: 20m (was: 10m)
> Rename compaction "attempted" status
>
>
> Key: HIVE-24586
> URL: https://issues.apache.org/jira/browse/HIVE-24586
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> A compaction with "attempted" status sounds like compactor tried to compact
> the table/partition and failed. In reality it means one of these:
> * the Initiator did not queue compaction because the number of previously
> failed compactions has passed a threshold
> * the Initiator did not queue compaction because of an error
> In both these cases the user is still able initiate compaction manually. This
> should be made clearer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=531957&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531957
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 16:29
Start Date: 06/Jan/21 16:29
Worklog Time Spent: 10m
Work Description: belugabehr opened a new pull request #1833:
URL: https://github.com/apache/hive/pull/1833
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531957)
Time Spent: 0.5h (was: 20m)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24592) Revert Hive-24550
[
https://issues.apache.org/jira/browse/HIVE-24592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259858#comment-17259858
]
John Sherman commented on HIVE-24592:
-
It might be nice if we could have a precommit check on the "passed tests" that
checked how stale the tag is and deny the commit if it was older than a few
days or a week (though some might find it annoying). I'm a little surprised
this situation doesn't happen more often.
> Revert Hive-24550
> -
>
> Key: HIVE-24592
> URL: https://issues.apache.org/jira/browse/HIVE-24592
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Broke the build.
>
> {code:none}
> [ERROR] COMPILATION ERROR :
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> symbol: class TxnDbUtil
> location: package org.apache.hadoop.hive.metastore.txn
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile
> (default-testCompile) on project hive-it-unit: Compilation failure
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> [ERROR] symbol: class TxnDbUtil
> [ERROR] location: package org.apache.hadoop.hive.metastore.txn
> [ERROR]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259857#comment-17259857
]
Stamatis Zampetakis commented on HIVE-24590:
The appender/files are created and closed based on the queryId. This queryId is
in the log4j context (usually in ThreadLocal variables) and can be present in
various threads. I suspect that the thread local is cleared on some threads but
not on others. If you are tracking the creation of the appenders you can see if
the same appender is created twice.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24592) Revert Hive-24550
[
https://issues.apache.org/jira/browse/HIVE-24592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-24592.
---
Resolution: Fixed
Done.
> Revert Hive-24550
> -
>
> Key: HIVE-24592
> URL: https://issues.apache.org/jira/browse/HIVE-24592
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Broke the build.
>
> {code:none}
> [ERROR] COMPILATION ERROR :
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> symbol: class TxnDbUtil
> location: package org.apache.hadoop.hive.metastore.txn
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile
> (default-testCompile) on project hive-it-unit: Compilation failure
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> [ERROR] symbol: class TxnDbUtil
> [ERROR] location: package org.apache.hadoop.hive.metastore.txn
> [ERROR]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=531955&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531955
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 16:28
Start Date: 06/Jan/21 16:28
Worklog Time Spent: 10m
Work Description: belugabehr closed pull request #1833:
URL: https://github.com/apache/hive/pull/1833
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531955)
Time Spent: 20m (was: 10m)
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24592) Revert Hive-24550
[
https://issues.apache.org/jira/browse/HIVE-24592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-24592:
-
> Revert Hive-24550
> -
>
> Key: HIVE-24592
> URL: https://issues.apache.org/jira/browse/HIVE-24592
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Broke the build.
>
> {code:none}
> [ERROR] COMPILATION ERROR :
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> symbol: class TxnDbUtil
> location: package org.apache.hadoop.hive.metastore.txn
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile
> (default-testCompile) on project hive-it-unit: Compilation failure
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> [ERROR] symbol: class TxnDbUtil
> [ERROR] location: package org.apache.hadoop.hive.metastore.txn
> [ERROR]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259845#comment-17259845
]
Eugene Chung commented on HIVE-24590:
-
> A different file but for the same query right?
It seems not. The query which is done normally and the closing of its session
on the same HttpHandler thread.
I think in the flow of closing session, logging of TezClient.close() is
creating the operation log.
{noformat}
at org.apache.log4j.Category.log(Category.java:445)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
at org.apache.tez.client.TezClient.stop(TezClient.java:737)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:710){noformat}
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24584) IndexOutOfBoundsException from Kryo when running msck repair
[
https://issues.apache.org/jira/browse/HIVE-24584?focusedWorklogId=531926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531926
]
ASF GitHub Bot logged work on HIVE-24584:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 15:43
Start Date: 06/Jan/21 15:43
Worklog Time Spent: 10m
Work Description: zeroflag commented on pull request #1828:
URL: https://github.com/apache/hive/pull/1828#issuecomment-755375883
cc: @shameersss1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531926)
Time Spent: 20m (was: 10m)
> IndexOutOfBoundsException from Kryo when running msck repair
>
>
> Key: HIVE-24584
> URL: https://issues.apache.org/jira/browse/HIVE-24584
> Project: Hive
> Issue Type: Bug
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The following exception is coming when running "msck repair table t1 sync
> partitions".
> {code:java}
> java.lang.IndexOutOfBoundsException: Index: 97, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[?:1.8.0_232]
> at java.util.ArrayList.get(ArrayList.java:433) ~[?:1.8.0_232]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:684)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:211)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectFromKryo(SerializationUtilities.java:814)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeExpressionFromKryo(SerializationUtilities.java:775)
> ~[hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.deserializeExpr(PartitionExpressionForMetastore.java:116)
> [hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartitionExpressionForMetastore.filterPartitionsByExpr(PartitionExpressionForMetastore.java:88)
> [hive-exec-3.1.3000.7.2.7.0-144.jar:3.1.3000.7.2.7.0-SNAPSHOT] {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-24591:
--
Description:
To make beeline as simple as possible, move its SLF4J logger implementation to
SLFJ-Simple logger. This will allow users to change the logging level simply
on the command line. Currently uses must create a Log4J configuration file
which is way too advance/cumbersome for a data analyst that just wants to use
SQL (and do some minor troubleshooting)
{code:none}
export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
beeline ...
{code}
http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
was:
To make beeline as simple as possible, move its SLF4J logger implementation to
SLFJ-Simple logger. This will allow users to change the logging level simply
on the command line. Currently uses must create a Log4J configuration file
which is way too advance/cumbersome for a data analyst that just wants to use
SQL (and do some minor troubleshooting)
{code:none}
beeline -Dorg.slf4j.simpleLogger.defaultLogLevel=debug ...
{code}
http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> export HADOOP_CLIENT_OPTS="-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
> beeline ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?focusedWorklogId=531911&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531911
]
ASF GitHub Bot logged work on HIVE-24591:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 15:14
Start Date: 06/Jan/21 15:14
Worklog Time Spent: 10m
Work Description: belugabehr opened a new pull request #1833:
URL: https://github.com/apache/hive/pull/1833
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531911)
Remaining Estimate: 0h
Time Spent: 10m
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> beeline -Dorg.slf4j.simpleLogger.defaultLogLevel=debug ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24591:
--
Labels: pull-request-available (was: )
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> beeline -Dorg.slf4j.simpleLogger.defaultLogLevel=debug ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24591) Move Beeline To SLF4J Simple Logger
[
https://issues.apache.org/jira/browse/HIVE-24591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-24591:
-
> Move Beeline To SLF4J Simple Logger
> ---
>
> Key: HIVE-24591
> URL: https://issues.apache.org/jira/browse/HIVE-24591
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> To make beeline as simple as possible, move its SLF4J logger implementation
> to SLFJ-Simple logger. This will allow users to change the logging level
> simply on the command line. Currently uses must create a Log4J configuration
> file which is way too advance/cumbersome for a data analyst that just wants
> to use SQL (and do some minor troubleshooting)
> {code:none}
> beeline -Dorg.slf4j.simpleLogger.defaultLogLevel=debug ...
> {code}
> http://www.slf4j.org/api/org/slf4j/impl/SimpleLogger.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259778#comment-17259778
]
Stamatis Zampetakis commented on HIVE-24590:
bq. Then it seems that operation log file with the different name is created by
the same thread
A different file but for the same query right?
After stopping the appender the log4j context should be cleared otherwise
whatever is logged afterwards (with the same context) is gonna recreate an
appender that we cannot close.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?focusedWorklogId=531898&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531898 ] ASF GitHub Bot logged work on HIVE-24550: - Author: ASF GitHub Bot Created on: 06/Jan/21 14:18 Start Date: 06/Jan/21 14:18 Worklog Time Spent: 10m Work Description: pvary merged pull request #1795: URL: https://github.com/apache/hive/pull/1795 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531898) Time Spent: 50m (was: 40m) > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-24550. --- Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the patch [~pvargacl]! > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: Peter Varga >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-15820) comment at the head of beeline -e
[
https://issues.apache.org/jira/browse/HIVE-15820?focusedWorklogId=531888&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531888
]
ASF GitHub Bot logged work on HIVE-15820:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 14:05
Start Date: 06/Jan/21 14:05
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #1814:
URL: https://github.com/apache/hive/pull/1814#discussion_r552640050
##
File path: beeline/src/test/org/apache/hive/beeline/TestCommands.java
##
@@ -30,21 +30,35 @@
@Test
public void testLinesEndingWithComments() {
-int[] escape = {-1};
-assertEquals("show tables;", removeComments("show tables;",escape));
-assertEquals("show tables;", removeComments("show tables;
--comments",escape));
-assertEquals("show tables;", removeComments("show tables;
---comments",escape));
-assertEquals("show tables;", removeComments("show tables;
---comments;one;two;three",escape));
-assertEquals("show", removeComments("show-- tables;
---comments",escape));
-assertEquals("show", removeComments("show --tables;
---comments",escape));
-assertEquals("s", removeComments("s--how --tables;
---comments",escape));
-assertEquals("", removeComments("-- show tables; ---comments",escape));
+assertEquals("show tables;", removeComments("show tables;"));
+assertEquals("show tables;", removeComments("show tables; --comments"));
+assertEquals("show tables;", removeComments("show tables;
---comments"));
+assertEquals("show tables;", removeComments("show tables;
---comments;one;two;three"));
+assertEquals("show", removeComments("show-- tables; ---comments"));
+assertEquals("show", removeComments("show --tables; ---comments"));
+assertEquals("s", removeComments("s--how --tables; ---comments"));
+assertEquals("", removeComments("-- show tables; ---comments"));
-assertEquals("\"show tables\"", removeComments("\"show tables\"
--comments",escape));
-assertEquals("\"show --comments tables\"", removeComments("\"show
--comments tables\" --comments",escape));
-assertEquals("\"'show --comments' tables\"", removeComments("\"'show
--comments' tables\" --comments",escape));
-assertEquals("'show --comments tables'", removeComments("'show --comments
tables' --comments",escape));
-assertEquals("'\"show --comments tables\"'", removeComments("'\"show
--comments tables\"' --comments",escape));
+assertEquals("\"show tables\"", removeComments("\"show tables\"
--comments"));
+assertEquals("\"show --comments tables\"", removeComments("\"show
--comments tables\" --comments"));
+assertEquals("\"'show --comments' tables\"", removeComments("\"'show
--comments' tables\" --comments"));
+assertEquals("'show --comments tables'", removeComments("'show --comments
tables' --comments"));
+assertEquals("'\"show --comments tables\"'", removeComments("'\"show
--comments tables\"' --comments"));
+
+assertEquals("show tables;", removeComments("--comments\nshow tables;"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
--comments"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
---comments"));
+assertEquals("show tables;", removeComments("--comments\nshow tables;
---comments;one;two;three"));
+assertEquals("show", removeComments("--comments\nshow-- tables;
---comments"));
+assertEquals("show", removeComments("--comments\nshow --tables;
---comments"));
+assertEquals("s", removeComments("--comments\ns--how --tables;
---comments"));
+assertEquals("", removeComments("--comments\n-- show tables;
---comments"));
+
+assertEquals("\"show tables\"", removeComments("--comments\n\"show
tables\" --comments"));
+assertEquals("\"show --comments tables\"",
removeComments("--comments\n\"show --comments tables\" --comments"));
+assertEquals("\"'show --comments' tables\"",
removeComments("--comments\n\"'show --comments' tables\" --comments"));
+assertEquals("'show --comments tables'", removeComments("--comments\n'show
--comments tables' --comments"));
+assertEquals("'\"show --comments tables\"'",
removeComments("--comments\n'\"show --comments tables\"' --comments"));
Review comment:
Could you please add a test scenario where a multiline query strings has
comments in between fragments of query? just as an example
"select col1,
--partitioned year column
year,
--partitioned month column
month,
--partitioned date column
date
from test_table
where
--for a particular user
username = 'foo';"
should return something equivalent to
"select col1, year, month, date from test_table where username = 'foo';"
-
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259725#comment-17259725
]
Eugene Chung edited comment on HIVE-24590 at 1/6/21, 1:58 PM:
--
{noformat}
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/***/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/***/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/irteam_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
at
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.log4j.Category.forcedLog(Category.java:357)
at org.apache.log4j.Category.log(Category.java:445)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
at org.apache.tez.client.TezClient.stop(TezClient.java:737)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:710)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:677)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.close(TezSessionPoolSession.java:111)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:354)
at
org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1765)
at
org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:761)
at
org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.H
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259725#comment-17259725
]
Eugene Chung commented on HIVE-24590:
-
{noformat}
[2021-01-06T22:35:48,371] INFO [HiveServer2-HttpHandler-Pool: Thread-101]
client.TezClient: Shutting down Tez Session,
sessionName=og-20210106222500-bc649f09-a16a-4295-ac16-f3e86883364b,
applicationId=application_1591769205146_5391722
2021-01-06 22:35:48,373 HiveServer2-HttpHandler-Pool: Thread-101 WARN
/***/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
for RandomAccessFileManager is created.
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender$DebugTrace:
/***/logs/hive_operation/edcd87dd-1fd5-4bdb-b29b-e125a05c1dfd/irteam_20210106223510_2bf8d74d-5347-4e56-a316-7a59a83f860f
at
org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
at
org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:439)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:434)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.log4j.Category.forcedLog(Category.java:357)
at org.apache.log4j.Category.log(Category.java:445)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
at org.apache.tez.client.TezClient.stop(TezClient.java:737)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:710)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:677)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.close(TezSessionPoolSession.java:111)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:354)
at
org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1765)
at
org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:761)
at
org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259686#comment-17259686
]
Eugene Chung edited comment on HIVE-24590 at 1/6/21, 12:54 PM:
---
[^add_debug_log_and_trace.patch] I added some debug logs like this patch to
HushableRandomAccessFileAppender. (Note that my
HushableRandomAccessFileAppender is patched with HIVE-22753)
!Screen Shot 2021-01-06 at 21.38.32.png|width=893,height=38!
The operation log is stopped with HushableRandomAccessFileAppender.stop().
Then it seems that operation log file with the different name is created by the
same thread.
!Screen Shot 2021-01-06 at 21.47.28.png|width=1249,height=30!
was (Author: euigeun_chung):
[^add_debug_log_and_trace.patch] I added some debug logs like this patch to
HushableRandomAccessFileAppender. (Note that my
HushableRandomAccessFileAppender is patched with HIVE-22753)
!Screen Shot 2021-01-06 at 21.38.32.png|width=893,height=38!
Then it seems that operation log file with the different name is created by the
same thread.
!Screen Shot 2021-01-06 at 21.47.28.png|width=1249,height=30!
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259686#comment-17259686
]
Eugene Chung edited comment on HIVE-24590 at 1/6/21, 12:50 PM:
---
[^add_debug_log_and_trace.patch] I added some debug logs like this patch to
HushableRandomAccessFileAppender. (Note that my
HushableRandomAccessFileAppender is patched with HIVE-22753)
!Screen Shot 2021-01-06 at 21.38.32.png|width=893,height=38!
Then it seems that operation log file with the different name is created by the
same thread.
!Screen Shot 2021-01-06 at 21.47.28.png|width=1249,height=30!
was (Author: euigeun_chung):
[^add_debug_log_and_trace.patch] I added some debug logs like this patch to
HushableRandomAccessFileAppender. (Note that my
HushableRandomAccessFileAppender is patched with
[HIVE-22753|https://issues.apache.org/jira/browse/HIVE-22753])
!Screen Shot 2021-01-06 at 21.38.32.png|width=1574,height=67!
Then it seems that operation log file with the different name is created by the
same thread.
!Screen Shot 2021-01-06 at 21.47.28.png|width=1457,height=35!
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259686#comment-17259686
]
Eugene Chung commented on HIVE-24590:
-
[^add_debug_log_and_trace.patch] I added some debug logs like this patch to
HushableRandomAccessFileAppender. (Note that my
HushableRandomAccessFileAppender is patched with
[HIVE-22753|https://issues.apache.org/jira/browse/HIVE-22753])
!Screen Shot 2021-01-06 at 21.38.32.png|width=1574,height=67!
Then it seems that operation log file with the different name is created by the
same thread.
!Screen Shot 2021-01-06 at 21.47.28.png|width=1457,height=35!
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-24590:
Attachment: Screen Shot 2021-01-06 at 21.47.28.png
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, Screen Shot 2021-01-06 at 21.47.28.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-24590:
Attachment: Screen Shot 2021-01-06 at 21.38.32.png
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png, Screen
> Shot 2021-01-06 at 21.38.32.png, add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-24590:
Attachment: add_debug_log_and_trace.patch
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png,
> add_debug_log_and_trace.patch
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24575) VectorGroupByOperator reusing keys can lead to wrong results
[
https://issues.apache.org/jira/browse/HIVE-24575?focusedWorklogId=531865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531865
]
ASF GitHub Bot logged work on HIVE-24575:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 12:39
Start Date: 06/Jan/21 12:39
Worklog Time Spent: 10m
Work Description: dengzhhu653 closed pull request #1822:
URL: https://github.com/apache/hive/pull/1822
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531865)
Time Spent: 1h (was: 50m)
> VectorGroupByOperator reusing keys can lead to wrong results
>
>
> Key: HIVE-24575
> URL: https://issues.apache.org/jira/browse/HIVE-24575
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A common sql like
> {code:java}
> select category as category, count(distinct maskdid) as uv from
> dwd_internal_inc_d group by category{code}
> can have a wrong result on the trunk, the result of column category can be
> confused and
> aggregate of distinct maskdid is also wrong.
> After some debugging, We find that the problem is caused by wrong
> byteStarts[i] when using it to copy the current keys to the reusable keys:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneral.java#L351-L362]
> The byteStarts[i] is always 0 due to Arrays.fill(byteStarts, 0); so it copies
> the range from 0 other then the real start index to len of the current keys
> to the reusable keys when clone.byteValues[i].length >= byteValues[i].length
> met, which results to the problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24575) VectorGroupByOperator reusing keys can lead to wrong results
[
https://issues.apache.org/jira/browse/HIVE-24575?focusedWorklogId=531864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531864
]
ASF GitHub Bot logged work on HIVE-24575:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 12:39
Start Date: 06/Jan/21 12:39
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #1822:
URL: https://github.com/apache/hive/pull/1822#issuecomment-755276656
Thanks much for the help and review, @mustafaiman. close it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531864)
Time Spent: 50m (was: 40m)
> VectorGroupByOperator reusing keys can lead to wrong results
>
>
> Key: HIVE-24575
> URL: https://issues.apache.org/jira/browse/HIVE-24575
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> A common sql like
> {code:java}
> select category as category, count(distinct maskdid) as uv from
> dwd_internal_inc_d group by category{code}
> can have a wrong result on the trunk, the result of column category can be
> confused and
> aggregate of distinct maskdid is also wrong.
> After some debugging, We find that the problem is caused by wrong
> byteStarts[i] when using it to copy the current keys to the reusable keys:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneral.java#L351-L362]
> The byteStarts[i] is always 0 due to Arrays.fill(byteStarts, 0); so it copies
> the range from 0 other then the real start index to len of the current keys
> to the reusable keys when clone.byteValues[i].length >= byteValues[i].length
> met, which results to the problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24585) NPE in VectorizedOrcAcidRowBatchReader if LLAP is used with IO disabled
[
https://issues.apache.org/jira/browse/HIVE-24585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259674#comment-17259674
]
Ádám Szita commented on HIVE-24585:
---
Committed to master. Thanks for the review [~pvary].
> NPE in VectorizedOrcAcidRowBatchReader if LLAP is used with IO disabled
> ---
>
> Key: HIVE-24585
> URL: https://issues.apache.org/jira/browse/HIVE-24585
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> NPE is thrown if LLAP mode is turned on and LLAP daemon executes a query on
> an ACID table if LLAP IO is disabled. Although this doesn't seem to be a very
> useful LLAP environment setup, we'll need to cover this edge case too.
> {code:java}
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:543)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:189)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:431)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> ... 26 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.getOrcTail(VectorizedOrcAcidRowBatchReader.java:680)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.findMinMaxKeys(VectorizedOrcAcidRowBatchReader.java:426)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:273)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:159)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:154)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2074)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:428)
> ... 27 more {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24585) NPE in VectorizedOrcAcidRowBatchReader if LLAP is used with IO disabled
[
https://issues.apache.org/jira/browse/HIVE-24585?focusedWorklogId=531862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531862
]
ASF GitHub Bot logged work on HIVE-24585:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 12:35
Start Date: 06/Jan/21 12:35
Worklog Time Spent: 10m
Work Description: szlta merged pull request #1830:
URL: https://github.com/apache/hive/pull/1830
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531862)
Time Spent: 20m (was: 10m)
> NPE in VectorizedOrcAcidRowBatchReader if LLAP is used with IO disabled
> ---
>
> Key: HIVE-24585
> URL: https://issues.apache.org/jira/browse/HIVE-24585
> Project: Hive
> Issue Type: Bug
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> NPE is thrown if LLAP mode is turned on and LLAP daemon executes a query on
> an ACID table if LLAP IO is disabled. Although this doesn't seem to be a very
> useful LLAP environment setup, we'll need to cover this edge case too.
> {code:java}
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:206)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:543)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:189)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> ... 15 more
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:431)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> ... 26 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.getOrcTail(VectorizedOrcAcidRowBatchReader.java:680)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.findMinMaxKeys(VectorizedOrcAcidRowBatchReader.java:426)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:273)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:159)
> at
> org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowBatchReader.(VectorizedOrcAcidRowBatchReader.java:154)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2074)
> at
> org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:428)
> ... 27 more {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24139) VectorGroupByOperator is not flushing hash table entries as needed
[ https://issues.apache.org/jira/browse/HIVE-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa İman updated HIVE-24139: Resolution: Duplicate Status: Resolved (was: Patch Available) > VectorGroupByOperator is not flushing hash table entries as needed > -- > > Key: HIVE-24139 > URL: https://issues.apache.org/jira/browse/HIVE-24139 > Project: Hive > Issue Type: Bug >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > After https://issues.apache.org/jira/browse/HIVE-23975 introduced a bug where > copyKey mutates some key wrappers while copying. This Jira is to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24575) VectorGroupByOperator reusing keys can lead to wrong results
[
https://issues.apache.org/jira/browse/HIVE-24575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa İman resolved HIVE-24575.
-
Resolution: Fixed
> VectorGroupByOperator reusing keys can lead to wrong results
>
>
> Key: HIVE-24575
> URL: https://issues.apache.org/jira/browse/HIVE-24575
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A common sql like
> {code:java}
> select category as category, count(distinct maskdid) as uv from
> dwd_internal_inc_d group by category{code}
> can have a wrong result on the trunk, the result of column category can be
> confused and
> aggregate of distinct maskdid is also wrong.
> After some debugging, We find that the problem is caused by wrong
> byteStarts[i] when using it to copy the current keys to the reusable keys:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneral.java#L351-L362]
> The byteStarts[i] is always 0 due to Arrays.fill(byteStarts, 0); so it copies
> the range from 0 other then the real start index to len of the current keys
> to the reusable keys when clone.byteValues[i].length >= byteValues[i].length
> met, which results to the problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24575) VectorGroupByOperator reusing keys can lead to wrong results
[
https://issues.apache.org/jira/browse/HIVE-24575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa İman updated HIVE-24575:
Fix Version/s: 4.0.0
> VectorGroupByOperator reusing keys can lead to wrong results
>
>
> Key: HIVE-24575
> URL: https://issues.apache.org/jira/browse/HIVE-24575
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A common sql like
> {code:java}
> select category as category, count(distinct maskdid) as uv from
> dwd_internal_inc_d group by category{code}
> can have a wrong result on the trunk, the result of column category can be
> confused and
> aggregate of distinct maskdid is also wrong.
> After some debugging, We find that the problem is caused by wrong
> byteStarts[i] when using it to copy the current keys to the reusable keys:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneral.java#L351-L362]
> The byteStarts[i] is always 0 due to Arrays.fill(byteStarts, 0); so it copies
> the range from 0 other then the real start index to len of the current keys
> to the reusable keys when clone.byteValues[i].length >= byteValues[i].length
> met, which results to the problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24575) VectorGroupByOperator reusing keys can lead to wrong results
[
https://issues.apache.org/jira/browse/HIVE-24575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259659#comment-17259659
]
Mustafa İman commented on HIVE-24575:
-
Merged to master. Thanks [~dengzh]
> VectorGroupByOperator reusing keys can lead to wrong results
>
>
> Key: HIVE-24575
> URL: https://issues.apache.org/jira/browse/HIVE-24575
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> A common sql like
> {code:java}
> select category as category, count(distinct maskdid) as uv from
> dwd_internal_inc_d group by category{code}
> can have a wrong result on the trunk, the result of column category can be
> confused and
> aggregate of distinct maskdid is also wrong.
> After some debugging, We find that the problem is caused by wrong
> byteStarts[i] when using it to copy the current keys to the reusable keys:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/wrapper/VectorHashKeyWrapperGeneral.java#L351-L362]
> The byteStarts[i] is always 0 due to Arrays.fill(byteStarts, 0); so it copies
> the range from 0 other then the real start index to len of the current keys
> to the reusable keys when clone.byteValues[i].length >= byteValues[i].length
> met, which results to the problem.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259649#comment-17259649
]
Stamatis Zampetakis commented on HIVE-24590:
I assume the reason of why there is still a leak is the same with HIVE-24569,
meaning that cleanup is performed with Operation#cleanupOperationLog so one
appender is closed but then another thread comes and attempts to write to the
same file thus creating an additional appender that will never be closed.
This also explains why the content of the respective log file has only one
line. During the {{cleanupOperationLog}}, {{OperationLogCleaner}} is called so
the full operation log file will be deleted but then if somebody else tries to
write to it (e.g., TezClient) the file will be recreated leading to the leak.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24571) Redundant code in SemanticAnalyzer.java
[ https://issues.apache.org/jira/browse/HIVE-24571?focusedWorklogId=531830&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531830 ] ASF GitHub Bot logged work on HIVE-24571: - Author: ASF GitHub Bot Created on: 06/Jan/21 11:30 Start Date: 06/Jan/21 11:30 Worklog Time Spent: 10m Work Description: zzccctv closed pull request #1819: URL: https://github.com/apache/hive/pull/1819 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531830) Time Spent: 1h 20m (was: 1h 10m) > Redundant code in SemanticAnalyzer.java > --- > > Key: HIVE-24571 > URL: https://issues.apache.org/jira/browse/HIVE-24571 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.3.7 >Reporter: zhangzhanchang >Assignee: zhangzhanchang >Priority: Minor > Labels: pull-request-available > Attachments: image-2020-12-27-16-42-59-875.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > In the newly released release-2.3.7, the file SemanticAnalyzer.java There are > redundant codes，Other branche-2 also have this problem > !image-2020-12-27-16-42-59-875.png|width=781,height=411! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-24590:
--
Assignee: Stamatis Zampetakis
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Assignee: Stamatis Zampetakis
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259616#comment-17259616
]
Stamatis Zampetakis commented on HIVE-24590:
[~euigeun_chung] OK I just noticed in the description that you are referring
to the HS2 process.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259615#comment-17259615
]
Eugene Chung commented on HIVE-24590:
-
[~zabetak] It's HS2.
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-24590:
Description:
I'm using Hive 3.1.2 with options below.
* hive.server2.logging.operation.enabled=true
* hive.server2.logging.operation.level=VERBOSE
* hive.async.log.enabled=false
I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128 but
HS2 still leaks log4j RandomAccessFileManager.
!Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
I checked the operation log file which is not closed/deleted properly.
!Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
Then there's the log,
{code:java}
client.TezClient: Shutting down Tez Session, sessionName= {code}
!Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
was:
I'm using Hive 3.1.2 with options below.
* hive.server2.logging.operation.enabled=true
* hive.server2.logging.operation.level=VERBOSE
* hive.async.log.enabled=false
I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128 but
it still leaks log4j RandomAccessFileManager.
!Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
I checked the operation log file which is not closed/deleted properly.
!Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
Then there's the log,
{code:java}
client.TezClient: Shutting down Tez Session, sessionName= {code}
!Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but HS2 still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259613#comment-17259613
]
Stamatis Zampetakis commented on HIVE-24590:
[~euigeun_chung] which process is leaking descriptors? HS2 or LLAP?
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but it still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24540) Add test hive shell for simpler execution tests and debugging
[ https://issues.apache.org/jira/browse/HIVE-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24540: -- Labels: pull-request-available (was: ) > Add test hive shell for simpler execution tests and debugging > - > > Key: HIVE-24540 > URL: https://issues.apache.org/jira/browse/HIVE-24540 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the Apache Iceberg project, we've been using a > TestHiveShell/TestHiveMetastore class for running query execution unit tests, > which made our life much easier both in terms of writing test and debugging > the code from an IDE. It would have value bringing it to the Apache Hive > project as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24540) Add test hive shell for simpler execution tests and debugging
[ https://issues.apache.org/jira/browse/HIVE-24540?focusedWorklogId=531816&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531816 ] ASF GitHub Bot logged work on HIVE-24540: - Author: ASF GitHub Bot Created on: 06/Jan/21 11:02 Start Date: 06/Jan/21 11:02 Worklog Time Spent: 10m Work Description: marton-bod closed pull request #1832: URL: https://github.com/apache/hive/pull/1832 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531816) Remaining Estimate: 0h Time Spent: 10m > Add test hive shell for simpler execution tests and debugging > - > > Key: HIVE-24540 > URL: https://issues.apache.org/jira/browse/HIVE-24540 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In the Apache Iceberg project, we've been using a > TestHiveShell/TestHiveMetastore class for running query execution unit tests, > which made our life much easier both in terms of writing test and debugging > the code from an IDE. It would have value bringing it to the Apache Hive > project as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24590) Operation Logging still leaks the log4j Appenders
[
https://issues.apache.org/jira/browse/HIVE-24590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Chung updated HIVE-24590:
Description:
I'm using Hive 3.1.2 with options below.
* hive.server2.logging.operation.enabled=true
* hive.server2.logging.operation.level=VERBOSE
* hive.async.log.enabled=false
I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128 but
it still leaks log4j RandomAccessFileManager.
!Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
I checked the operation log file which is not closed/deleted properly.
!Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
Then there's the log,
{code:java}
client.TezClient: Shutting down Tez Session, sessionName= {code}
!Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
was:
I'm using Hive 3.1.2 with options below.
* hive.server2.logging.operation.enabled=true
* hive.server2.logging.operation.level=VERBOSE
* hive.async.log.enabled=false
I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128 but
it still leaks log4j RandomAccessFileManager.
!Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
I checked the operation log file which is not closed.
!Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
Then there's the log,
{code:java}
client.TezClient: Shutting down Tez Session, sessionName= {code}
!Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
> Operation Logging still leaks the log4j Appenders
> -
>
> Key: HIVE-24590
> URL: https://issues.apache.org/jira/browse/HIVE-24590
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: Eugene Chung
>Priority: Major
> Attachments: Screen Shot 2021-01-06 at 18.42.05.png, Screen Shot
> 2021-01-06 at 18.42.24.png, Screen Shot 2021-01-06 at 18.42.55.png
>
>
> I'm using Hive 3.1.2 with options below.
> * hive.server2.logging.operation.enabled=true
> * hive.server2.logging.operation.level=VERBOSE
> * hive.async.log.enabled=false
> I already know the ticket, https://issues.apache.org/jira/browse/HIVE-17128
> but it still leaks log4j RandomAccessFileManager.
> !Screen Shot 2021-01-06 at 18.42.05.png|width=756,height=197!
> I checked the operation log file which is not closed/deleted properly.
> !Screen Shot 2021-01-06 at 18.42.24.png|width=603,height=272!
> Then there's the log,
> {code:java}
> client.TezClient: Shutting down Tez Session, sessionName= {code}
> !Screen Shot 2021-01-06 at 18.42.55.png|width=1372,height=26!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23553) Upgrade ORC version to 1.6.6
[ https://issues.apache.org/jira/browse/HIVE-23553?focusedWorklogId=531806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531806 ] ASF GitHub Bot logged work on HIVE-23553: - Author: ASF GitHub Bot Created on: 06/Jan/21 10:38 Start Date: 06/Jan/21 10:38 Worklog Time Spent: 10m Work Description: pgaref edited a comment on pull request #1823: URL: https://github.com/apache/hive/pull/1823#issuecomment-754556917 > note: I tend to use distinct version numbers because snapshots might get cached and not updated - but that was an issue with the old ptest infra; I guess the current setup will handle that better... > > anything will do which can serve a web page - I wanted to add https://raw.githubusercontent.com/pgaref/mave-repo/main/ it for you - however that page returns error 400 for everything... Hey Zoltan -- I noticed the 400 myself for listings and non-existing files (GitHub policy?) but maven pulling seems to work, for example, check https://raw.githubusercontent.com/pgaref/mave-repo/main/org/apache/orc/orc-core/maven-metadata-local.xml To be on the safe side though I created this Repsy public repo: https://repo.repsy.io/mvn/pgaref/repository Feel free to add this instead :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531806) Time Spent: 2h (was: 1h 50m) > Upgrade ORC version to 1.6.6 > > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Apache Hive is currently on 1.5.X version and in order to take advantage of > the latest ORC improvements such as column encryption we have to bump to > 1.6.X. > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288&styleName=&projectId=12318320&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin > Even though ORC reader could work out of the box, HIVE LLAP is heavily > depending on internal ORC APIs e.g., to retrieve and store File Footers, > Tails, streams – un/compress RG data etc. As there ware many internal changes > from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the > upgrade is not straightforward. > This Umbrella Jira tracks this upgrade effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-17128) Operation Logging leaks file descriptors as the log4j Appender is never closed
[ https://issues.apache.org/jira/browse/HIVE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259585#comment-17259585 ] Eugene Chung commented on HIVE-17128: - [~zabetak] https://issues.apache.org/jira/browse/HIVE-24590 > Operation Logging leaks file descriptors as the log4j Appender is never closed > -- > > Key: HIVE-17128 > URL: https://issues.apache.org/jira/browse/HIVE-17128 > Project: Hive > Issue Type: Bug > Components: Logging >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17128.1.patch, HIVE-17128.2.patch, > HIVE-17128.3.patch > > > [HIVE-16061] and [HIVE-16400] changed operation logging to use the Log4j2 > RoutingAppender to automatically output the log for each query into each > individual operation log file. As log4j does not know when a query is > finished it keeps the OutputStream in the Appender open even when the query > completes. The stream holds a file descriptor and so we leak file > descriptors. Note that we are already careful to close any streams reading > from the operation log file. > h2. Fix > To fix this we use a technique described in the comments of [LOG4J2-510] > which uses reflection to close the appender. The test in > TestOperationLoggingLayout will be extended to check that the Appender is > closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24511) Fix typo in SerDeStorageSchemaReader
[
https://issues.apache.org/jira/browse/HIVE-24511?focusedWorklogId=531802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531802
]
ASF GitHub Bot logged work on HIVE-24511:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 10:35
Start Date: 06/Jan/21 10:35
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on a change in pull request #1757:
URL: https://github.com/apache/hive/pull/1757#discussion_r552419455
##
File path:
metastore/src/java/org/apache/hadoop/hive/metastore/SerDeStorageSchemaReader.java
##
@@ -47,10 +48,10 @@
Deserializer s = HiveMetaStoreUtils.getDeserializer(conf, tbl, false);
return HiveMetaStoreUtils.getFieldsFromDeserializer(tbl.getTableName(),
s);
} catch (Exception e) {
- StringUtils.stringifyException(e);
- throw new MetaException(e.getMessage());
+ throw new MetaException(StringUtils.stringifyException(e));
Review comment:
Hi, @belugabehr could you please take another look at the changes? thank
you
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531802)
Time Spent: 50m (was: 40m)
> Fix typo in SerDeStorageSchemaReader
>
>
> Key: HIVE-24511
> URL: https://issues.apache.org/jira/browse/HIVE-24511
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> 1, Close the created classloader to release resources.
> 2, More detail error messages on MetaException when throwing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24541) Add config to set a default storage handler class
[ https://issues.apache.org/jira/browse/HIVE-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-24541. --- Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks for the patch [~Marton Bod]! > Add config to set a default storage handler class > - > > Key: HIVE-24541 > URL: https://issues.apache.org/jira/browse/HIVE-24541 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Add a config param "hive.default.storage.handler.class" so we can set a > default storage handler class that can be used for all create table > statements. By default it would be an empty string, taking no effect. > This would allow existing user queries to be reused for a new table format > for example, such as Iceberg. > For example, after setting in the config: hive.default.storage.handler.class= > org.apache.iceberg.mr.hive.HiveIcebergStorageHandler > The query: CREATE TABLE abc (a int, b string) LOCATION ... > would be equivalent to: CREATE TABLE abc (a int, b string) STORED BY > 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24541) Add config to set a default storage handler class
[ https://issues.apache.org/jira/browse/HIVE-24541?focusedWorklogId=531797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531797 ] ASF GitHub Bot logged work on HIVE-24541: - Author: ASF GitHub Bot Created on: 06/Jan/21 10:26 Start Date: 06/Jan/21 10:26 Worklog Time Spent: 10m Work Description: pvary merged pull request #1797: URL: https://github.com/apache/hive/pull/1797 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531797) Time Spent: 1h (was: 50m) > Add config to set a default storage handler class > - > > Key: HIVE-24541 > URL: https://issues.apache.org/jira/browse/HIVE-24541 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Add a config param "hive.default.storage.handler.class" so we can set a > default storage handler class that can be used for all create table > statements. By default it would be an empty string, taking no effect. > This would allow existing user queries to be reused for a new table format > for example, such as Iceberg. > For example, after setting in the config: hive.default.storage.handler.class= > org.apache.iceberg.mr.hive.HiveIcebergStorageHandler > The query: CREATE TABLE abc (a int, b string) LOCATION ... > would be equivalent to: CREATE TABLE abc (a int, b string) STORED BY > 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION ... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-17128) Operation Logging leaks file descriptors as the log4j Appender is never closed
[ https://issues.apache.org/jira/browse/HIVE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259577#comment-17259577 ] Eugene Chung commented on HIVE-17128: - [~zabetak] I forgot to make a ticket at that time. Now I'm preparing for it. > Operation Logging leaks file descriptors as the log4j Appender is never closed > -- > > Key: HIVE-17128 > URL: https://issues.apache.org/jira/browse/HIVE-17128 > Project: Hive > Issue Type: Bug > Components: Logging >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17128.1.patch, HIVE-17128.2.patch, > HIVE-17128.3.patch > > > [HIVE-16061] and [HIVE-16400] changed operation logging to use the Log4j2 > RoutingAppender to automatically output the log for each query into each > individual operation log file. As log4j does not know when a query is > finished it keeps the OutputStream in the Appender open even when the query > completes. The stream holds a file descriptor and so we leak file > descriptors. Note that we are already careful to close any streams reading > from the operation log file. > h2. Fix > To fix this we use a technique described in the comments of [LOG4J2-510] > which uses reflection to close the appender. The test in > TestOperationLoggingLayout will be extended to check that the Appender is > closed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24519) Optimize MV: Materialized views should not rebuild when tables are not modified
[
https://issues.apache.org/jira/browse/HIVE-24519?focusedWorklogId=531794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531794
]
ASF GitHub Bot logged work on HIVE-24519:
-
Author: ASF GitHub Bot
Created on: 06/Jan/21 10:16
Start Date: 06/Jan/21 10:16
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #1772:
URL: https://github.com/apache/hive/pull/1772#discussion_r552486656
##
File path:
ql/src/test/results/clientnegative/materialized_view_authorization_rebuild_no_grant.q.out
##
@@ -33,4 +33,4 @@ POSTHOOK: type: CREATE_MATERIALIZED_VIEW
POSTHOOK: Input: default@amvrng_table
POSTHOOK: Output: database:default
POSTHOOK: Output: default@amvrng_mat_view
-FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation QUERY [[INSERT,
DELETE] on Object [type=TABLE_OR_VIEW, name=default.amvrng_mat_view,
action=INSERT_OVERWRITE]]
+FAILED: HiveAccessControlException Permission denied: Principal [name=user1,
type=USER] does not have following privileges for operation
ALTER_MATERIALIZED_VIEW_REBUILD [[OBJECT OWNERSHIP] on Object
[type=TABLE_OR_VIEW, name=default.amvrng_mat_view, action=INSERT_OVERWRITE]]
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 531794)
Time Spent: 2h 20m (was: 2h 10m)
> Optimize MV: Materialized views should not rebuild when tables are not
> modified
> ---
>
> Key: HIVE-24519
> URL: https://issues.apache.org/jira/browse/HIVE-24519
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Reporter: Rajesh Balamohan
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> e.g
> {noformat}
> create materialized view c_c_address as
> select c_customer_sk from customer c, customer_address ca where
> c_current_addr_sk = ca.ca_address_id;
> ALTER MATERIALIZED VIEW c_c_address REBUILD; <-- This shouldn't trigger
> rebuild, when source tables are not modified
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24337) Cache delete delta files in LLAP cache
[ https://issues.apache.org/jira/browse/HIVE-24337?focusedWorklogId=531781&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531781 ] ASF GitHub Bot logged work on HIVE-24337: - Author: ASF GitHub Bot Created on: 06/Jan/21 09:41 Start Date: 06/Jan/21 09:41 Worklog Time Spent: 10m Work Description: szlta commented on pull request #1776: URL: https://github.com/apache/hive/pull/1776#issuecomment-755194322 Tested with hive.llap.io.cache.deletedeltas=none. All tests passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 531781) Time Spent: 4h 40m (was: 4.5h) > Cache delete delta files in LLAP cache > -- > > Key: HIVE-24337 > URL: https://issues.apache.org/jira/browse/HIVE-24337 > Project: Hive > Issue Type: New Feature >Reporter: Ádám Szita >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > HIVE-23824 added the functionality of caching metadata part of orc files in > LLAP cache, so that ACID reads can be faster. However the content itself > still needs to be read in every single time. If this could be cached too, > additional time could be saved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
