[jira] [Work logged] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?focusedWorklogId=536724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536724
]
ASF GitHub Bot logged work on HIVE-24569:
-
Author: ASF GitHub Bot
Created on: 16/Jan/21 06:35
Start Date: 16/Jan/21 06:35
Worklog Time Spent: 10m
Work Description: prasanthj commented on a change in pull request #1858:
URL: https://github.com/apache/hive/pull/1858#discussion_r558815600
##
File path: llap-server/src/main/resources/llap-daemon-log4j2.properties
##
@@ -81,23 +81,21 @@ appender.query-routing.type = Routing
appender.query-routing.name = query-routing
appender.query-routing.routes.type = Routes
appender.query-routing.routes.pattern = $${ctx:queryId}
-#Purge polciy for query-based Routing Appender
-appender.query-routing.purgePolicy.type = LlapRoutingAppenderPurgePolicy
-# Note: Do not change this name without changing the corresponding entry in
LlapConstants
-appender.query-routing.purgePolicy.name = llapLogPurgerQueryRouting
+#Purge policy for query-based Routing Appender
+appender.query-routing.purgePolicy.type = IdlePurgePolicy
+appender.query-routing.purgePolicy.timeToLive = 5
Review comment:
Yes. 5 sec is too low. I would keep give it atleast 30 or 60s. Also I
remember the log router will rename the files with .done suffix to mark the
completion of query and hence the log file. Does it still happen? If we no
longer follow the filename convention it will be worth checking if other
consumers are expecting files with .done suffix (yarn log aggregation, twz ui)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536724)
Time Spent: 50m (was: 40m)
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_160865912
[jira] [Commented] (HIVE-24606) Multi-stage materialized CTEs can lose intermediate data
[
https://issues.apache.org/jira/browse/HIVE-24606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266507#comment-17266507
]
okumin commented on HIVE-24606:
---
CI has passed. This PR would require review by a SemanticAnalyzer expert.
https://github.com/apache/hive/pull/1873
> Multi-stage materialized CTEs can lose intermediate data
>
>
> Key: HIVE-24606
> URL: https://issues.apache.org/jira/browse/HIVE-24606
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.3.7, 3.1.2, 4.0.0
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> With complex multi-stage CTEs, Hive can start a latter stage before its
> previous stage finishes.
> That's because `SemanticAnalyzer#toRealRootTasks` can fail to resolve
> dependency between multistage materialized CTEs when a non-materialized CTE
> cuts in.
>
> [https://github.com/apache/hive/blob/425e1ff7c054f87c4db87e77d004282d529599ae/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1414]
>
> For example, when submitting this query,
> {code:sql}
> SET hive.optimize.cte.materialize.threshold=2;
> SET hive.optimize.cte.materialize.full.aggregate.only=false;
> WITH x AS ( SELECT 'x' AS id ), -- not materialized
> a1 AS ( SELECT 'a1' AS id ), -- materialized by a2 and the root
> a2 AS ( SELECT 'a2 <- ' || id AS id FROM a1) -- materialized by the root
> SELECT * FROM a1
> UNION ALL
> SELECT * FROM x
> UNION ALL
> SELECT * FROM a2
> UNION ALL
> SELECT * FROM a2;
> {code}
> `toRealRootTask` will traverse the CTEs in order of `a1`, `x`, and `a2`. It
> means the dependency between `a1` and `a2` will be ignored and `a2` can start
> without waiting for `a1`. As a result, the above query returns the following
> result.
> {code:java}
> +-+
> | id |
> +-+
> | a1 |
> | x |
> +-+
> {code}
> For your information, I ran this test with revision =
> 425e1ff7c054f87c4db87e77d004282d529599ae.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24327) AtlasServer entity may not be present during first Atlas metadata dump
[ https://issues.apache.org/jira/browse/HIVE-24327?focusedWorklogId=536696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536696 ] ASF GitHub Bot logged work on HIVE-24327: - Author: ASF GitHub Bot Created on: 16/Jan/21 01:32 Start Date: 16/Jan/21 01:32 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1623: URL: https://github.com/apache/hive/pull/1623 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536696) Time Spent: 1h 10m (was: 1h) > AtlasServer entity may not be present during first Atlas metadata dump > -- > > Key: HIVE-24327 > URL: https://issues.apache.org/jira/browse/HIVE-24327 > Project: Hive > Issue Type: Bug >Reporter: Pravin Sinha >Assignee: Pravin Sinha >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24327.01.patch, HIVE-24327.02.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24273) grouping key is case sensitive
[ https://issues.apache.org/jira/browse/HIVE-24273?focusedWorklogId=536694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536694 ] ASF GitHub Bot logged work on HIVE-24273: - Author: ASF GitHub Bot Created on: 16/Jan/21 01:32 Start Date: 16/Jan/21 01:32 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1579: URL: https://github.com/apache/hive/pull/1579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536694) Time Spent: 50m (was: 40m) > grouping key is case sensitive > --- > > Key: HIVE-24273 > URL: https://issues.apache.org/jira/browse/HIVE-24273 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 4.0.0 >Reporter: zhaolong >Assignee: zhaolong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: 0001-fix-HIVE-24273-grouping-key-is-case-sensitive.patch > > Time Spent: 50m > Remaining Estimate: 0h > > grouping key is case sensitive, the follow step can reproduce > 1.create table testaa(name string, age int); > 2.select GROUPING(name) from testaa group by name; -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24330) Automate setting permissions on cmRoot directories.
[ https://issues.apache.org/jira/browse/HIVE-24330?focusedWorklogId=536695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536695 ] ASF GitHub Bot logged work on HIVE-24330: - Author: ASF GitHub Bot Created on: 16/Jan/21 01:32 Start Date: 16/Jan/21 01:32 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1630: URL: https://github.com/apache/hive/pull/1630 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536695) Time Spent: 1h 20m (was: 1h 10m) > Automate setting permissions on cmRoot directories. > --- > > Key: HIVE-24330 > URL: https://issues.apache.org/jira/browse/HIVE-24330 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24330.01.patch, HIVE-24330.02.patch, > HIVE-24330.03.patch, HIVE-24330.04.patch, HIVE-24330.05.patch, > HIVE-24330.06.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24390) Spelling
[ https://issues.apache.org/jira/browse/HIVE-24390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24390: -- Labels: pull-request-available (was: ) > Spelling > > > Key: HIVE-24390 > URL: https://issues.apache.org/jira/browse/HIVE-24390 > Project: Hive > Issue Type: Bug >Reporter: Josh Soref >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24390) Spelling
[ https://issues.apache.org/jira/browse/HIVE-24390?focusedWorklogId=536693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536693 ] ASF GitHub Bot logged work on HIVE-24390: - Author: ASF GitHub Bot Created on: 16/Jan/21 01:31 Start Date: 16/Jan/21 01:31 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1674: URL: https://github.com/apache/hive/pull/1674#issuecomment-761285599 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536693) Remaining Estimate: 0h Time Spent: 10m > Spelling > > > Key: HIVE-24390 > URL: https://issues.apache.org/jira/browse/HIVE-24390 > Project: Hive > Issue Type: Bug >Reporter: Josh Soref >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24647) Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
[
https://issues.apache.org/jira/browse/HIVE-24647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Wood updated HIVE-24647:
---
Description:
A scenario has been discovered that seems to result in attempts to vectorize
and run non-vectorizable queries. The conditions of this scenario are as
follows:A scenario has been discovered that seems to result in attempts to
vectorize and run non-vectorizable queries. The conditions of this scenario are
as follows:
# The query {{SELECT}} s non-primitive type columns. I have tested this with
{{ARRAY}} and {{RECORD}} types.
# The query involves {{JOIN}} ing of two or more tables.
# The tables are external, and they use the same storage type. This may occur
with internal tables as well, but I am unable to test this in the Hive instance
I have access to. The storage types I have tested this with are Parquet and
Avro. If all tables use Parquet or all use Avro, the error is raised. If there
is a mixture of storage types, the query runs successfully.
I tried to dig through code to find and fix the issue myself, but unfortunately
I did not have a complete enough understanding of Hive and its dependencies to
pinpoint the issue.
Further examples and explanation - given the following two table definitions:
{code:sql}
CREATE EXTERNAL TABLE experimental.join_test (
int_col BIGINT,
array_col ARRAY ,
record_col STRUCT
)
STORED AS AVRO
LOCATION 's3://s3-bucket/join_test/'
TBLPROPERTIES (
'avro.schema.literal'='{"fields": [{"name": "int_col", "type": ["null",
"long"]},
{"name": "array_col",
"type": ["null", {"items": ["null", "long"], "type": "array"}]},
{"name": "record_col",
"type": ["null",
{"fields": [{"name": "a", "type": ["null", "long"]},
{"name": "b", "type": ["null", "long"]}],
"name": "record_col_0",
"type": "record"}]}],
"name": "Root",
"type": "record"}'
)
{code}
{code:sql}
CREATE EXTERNAL TABLE experimental.join_test2 (
int_col2 BIGINT,
str_col STRING
)
STORED AS AVRO
LOCATION 's3://s3-bucket/join_test2/'
TBLPROPERTIES (
'avro.schema.literal'='{"fields": [{"name": "int_col2", "type": ["null",
"long"]},
{"name": "str_col", "type": ["null", "string"]}],
"name": "Root",
"type": "record"}'
)
{code}
I can successfully query both of these tables, running the following queries:
{code:sql}
SELECT * FROM experimental.join_test;
1 [1,2] {"a":1,"b":2}
2 [3,4] {"a":3,"b":4}
3 [5,6] {"a":5,"b":6}
{code}
{code:sql}
SELECT * FROM experimental.join_test2;
1 should-appear-after-joining
2 should-appear-after-joining
3 should-appear-after-joining
4 shouldnt-appear-after-joining
{code}
I can also join the tables together, so long as I do not select the complex
type columns:
{code:sql}
SELECT int_col, int_col2, str_col
FROM experimental.join_test AS jt
JOIN experimental.join_test2 AS jt2 ON jt.int_col = jt2.int_col2;
3 3 should-appear-after-joining
2 2 should-appear-after-joining
1 1 should-appear-after-joining
{code}
But as soon as complex columns are introduced to the {{SELECT}} clause, an
error arises.
{code:sql}
SELECT *
FROM experimental.george_test AS jt
JOIN experimental.george_test2 AS jt2 ON jt.int_col = jt2.int_col2;
{code}
I will attach the full stack trace as a file - it is rather large. The
lowest-level error message given is slightly different depending on which
complex type is involved in raising the error.
* If it is an {{ARRAY}} type, the error message is
{{org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector}}
* If it is a {{RECORD}} type, the error message is
{{org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector}}
If the tables are of different types - for example, if the table {{join_test}}
was stored as Parquet instead of Avro - the query executes without issue.
From what I can tell by looking at the stack trace, Hive is attempting to run
this query as a vectorized query, leading to the error.Some further points of
support for this idea:
# Vectorization is not supported in queries that involved complex type
columns, which lines up with the fact that queries only fail when complex type
columns are included in the query.
# Explicitly setting {{hive.vectorized.execution.enabled}} to {{'false'}}
allows these queries to execute and finish normally.
was:
A scenario has been discovered that seems to result in attempts to vectorize
and run non-vectorizable queries. The conditions of this scenario are as
follows:A scenario has been discovered that seems to result in a
[jira] [Updated] (HIVE-24647) Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
[
https://issues.apache.org/jira/browse/HIVE-24647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Wood updated HIVE-24647:
---
Environment:
Hadoop and Hive instances running via Amazon EMR.
Hadoop version {{3.2.1}}
Hive version {{3.1.2}
Queries run via {{hive}} CLI.
OS: Amazon Linux 2
was:
Hadoop and Hive instances running via Amazon EMR.
Queries run via {{hive}} CLI.
OS: Amazon Linux 2
> Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
> -
>
> Key: HIVE-24647
> URL: https://issues.apache.org/jira/browse/HIVE-24647
> Project: Hive
> Issue Type: Bug
> Components: File Formats, hpl/sql, Query Processor, Vectorization
>Affects Versions: 3.1.2
> Environment: Hadoop and Hive instances running via Amazon EMR.
> Hadoop version {{3.2.1}}
> Hive version {{3.1.2}
> Queries run via {{hive}} CLI.
> OS: Amazon Linux 2
>Reporter: George Wood
>Priority: Major
> Attachments: hive_complex_join_err.txt
>
>
> A scenario has been discovered that seems to result in attempts to vectorize
> and run non-vectorizable queries. The conditions of this scenario are as
> follows:A scenario has been discovered that seems to result in attempts to
> vectorize and run non-vectorizable queries. The conditions of this scenario
> are as follows:
> # 1. The query {{SELECT}}s non-primitive type columns. I have tested this
> with {{ARRAY}} and {{RECORD}} types.
> # 2. The query involves {{JOIN}}ing of two or more tables.
> # 3. The tables are external, and they use the same storage type. This may
> occur with internal tables as well, but I am unable to test this in the Hive
> instance I have access to. The storage types I have tested this with are
> Parquet and Avro. If all tables use Parquet or all use Avro, the error is
> raised. If there is a mixture of storage types, the query runs successfully.
> I tried to dig through code to find and fix the issue myself, but
> unfortunately I did not have a complete enough understanding of Hive and its
> dependencies to pinpoint the issue.
> Further examples and explanation - given the following two table definitions:
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test (
> int_col BIGINT,
> array_col ARRAY ,
> record_col STRUCT
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col", "type": ["null",
> "long"]},
> {"name": "array_col",
> "type": ["null", {"items": ["null", "long"], "type": "array"}]},
> {"name": "record_col",
> "type": ["null",
> {"fields": [{"name": "a", "type": ["null", "long"]},
> {"name": "b", "type": ["null", "long"]}],
>"name": "record_col_0",
>"type": "record"}]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test2 (
> int_col2 BIGINT,
> str_col STRING
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test2/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col2", "type": ["null",
> "long"]},
> {"name": "str_col", "type": ["null", "string"]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> I can successfully query both of these tables, running the following queries:
> {code:sql}
> SELECT * FROM experimental.join_test;
> 1 [1,2] {"a":1,"b":2}
> 2 [3,4] {"a":3,"b":4}
> 3 [5,6] {"a":5,"b":6}
> {code}
> {code:sql}
> SELECT * FROM experimental.join_test2;
> 1 should-appear-after-joining
> 2 should-appear-after-joining
> 3 should-appear-after-joining
> 4 shouldnt-appear-after-joining
> {code}
> I can also join the tables together, so long as I do not select the complex
> type columns:
> {code:sql}
> SELECT int_col, int_col2, str_col
> FROM experimental.join_test AS jt
> JOIN experimental.join_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> 3 3 should-appear-after-joining
> 2 2 should-appear-after-joining
> 1 1 should-appear-after-joining
> {code}
> But as soon as complex columns are introduced to the {{SELECT}} clause, an
> error arises.
> {code:sql}
> SELECT *
> FROM experimental.george_test AS jt
> JOIN experimental.george_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> {code}
> I will attach the full stack trace as a file - it is rather large. The
> lowest-level error message given is slightly different depending on which
> complex type is involved in raising the error.
> * If it is an {{ARRAY}} type, the error message is
> {{org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2
[jira] [Updated] (HIVE-24647) Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
[
https://issues.apache.org/jira/browse/HIVE-24647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Wood updated HIVE-24647:
---
Environment:
Hadoop and Hive instances running via Amazon EMR.
Hadoop version {{3.2.1}}
Hive version {{3.1.2}}
Queries run via {{hive}} CLI.
OS: Amazon Linux 2
was:
Hadoop and Hive instances running via Amazon EMR.
Hadoop version {{3.2.1}}
Hive version {{3.1.2}
Queries run via {{hive}} CLI.
OS: Amazon Linux 2
> Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
> -
>
> Key: HIVE-24647
> URL: https://issues.apache.org/jira/browse/HIVE-24647
> Project: Hive
> Issue Type: Bug
> Components: File Formats, hpl/sql, Query Processor, Vectorization
>Affects Versions: 3.1.2
> Environment: Hadoop and Hive instances running via Amazon EMR.
> Hadoop version {{3.2.1}}
> Hive version {{3.1.2}}
> Queries run via {{hive}} CLI.
> OS: Amazon Linux 2
>Reporter: George Wood
>Priority: Major
> Attachments: hive_complex_join_err.txt
>
>
> A scenario has been discovered that seems to result in attempts to vectorize
> and run non-vectorizable queries. The conditions of this scenario are as
> follows:A scenario has been discovered that seems to result in attempts to
> vectorize and run non-vectorizable queries. The conditions of this scenario
> are as follows:
> # 1. The query {{SELECT}}s non-primitive type columns. I have tested this
> with {{ARRAY}} and {{RECORD}} types.
> # 2. The query involves {{JOIN}}ing of two or more tables.
> # 3. The tables are external, and they use the same storage type. This may
> occur with internal tables as well, but I am unable to test this in the Hive
> instance I have access to. The storage types I have tested this with are
> Parquet and Avro. If all tables use Parquet or all use Avro, the error is
> raised. If there is a mixture of storage types, the query runs successfully.
> I tried to dig through code to find and fix the issue myself, but
> unfortunately I did not have a complete enough understanding of Hive and its
> dependencies to pinpoint the issue.
> Further examples and explanation - given the following two table definitions:
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test (
> int_col BIGINT,
> array_col ARRAY ,
> record_col STRUCT
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col", "type": ["null",
> "long"]},
> {"name": "array_col",
> "type": ["null", {"items": ["null", "long"], "type": "array"}]},
> {"name": "record_col",
> "type": ["null",
> {"fields": [{"name": "a", "type": ["null", "long"]},
> {"name": "b", "type": ["null", "long"]}],
>"name": "record_col_0",
>"type": "record"}]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test2 (
> int_col2 BIGINT,
> str_col STRING
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test2/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col2", "type": ["null",
> "long"]},
> {"name": "str_col", "type": ["null", "string"]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> I can successfully query both of these tables, running the following queries:
> {code:sql}
> SELECT * FROM experimental.join_test;
> 1 [1,2] {"a":1,"b":2}
> 2 [3,4] {"a":3,"b":4}
> 3 [5,6] {"a":5,"b":6}
> {code}
> {code:sql}
> SELECT * FROM experimental.join_test2;
> 1 should-appear-after-joining
> 2 should-appear-after-joining
> 3 should-appear-after-joining
> 4 shouldnt-appear-after-joining
> {code}
> I can also join the tables together, so long as I do not select the complex
> type columns:
> {code:sql}
> SELECT int_col, int_col2, str_col
> FROM experimental.join_test AS jt
> JOIN experimental.join_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> 3 3 should-appear-after-joining
> 2 2 should-appear-after-joining
> 1 1 should-appear-after-joining
> {code}
> But as soon as complex columns are introduced to the {{SELECT}} clause, an
> error arises.
> {code:sql}
> SELECT *
> FROM experimental.george_test AS jt
> JOIN experimental.george_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> {code}
> I will attach the full stack trace as a file - it is rather large. The
> lowest-level error message given is slightly different depending on which
> complex type is involved in raising the error.
> * If it is an {{ARRAY}} type, the error message is
> {{org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
>
[jira] [Updated] (HIVE-24647) Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
[
https://issues.apache.org/jira/browse/HIVE-24647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Wood updated HIVE-24647:
---
Environment:
Hadoop and Hive instances running via Amazon EMR.
Queries run via {{hive}} CLI.
OS: Amazon Linux 2
was:
Hadoop and Hive instances running via Amazon EMR.
Queries run via `hive` CLI.
OS: Amazon Linux 2
> Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
> -
>
> Key: HIVE-24647
> URL: https://issues.apache.org/jira/browse/HIVE-24647
> Project: Hive
> Issue Type: Bug
> Components: File Formats, hpl/sql, Query Processor, Vectorization
>Affects Versions: 3.1.2
> Environment: Hadoop and Hive instances running via Amazon EMR.
> Queries run via {{hive}} CLI.
> OS: Amazon Linux 2
>Reporter: George Wood
>Priority: Major
> Attachments: hive_complex_join_err.txt
>
>
> A scenario has been discovered that seems to result in attempts to vectorize
> and run non-vectorizable queries. The conditions of this scenario are as
> follows:A scenario has been discovered that seems to result in attempts to
> vectorize and run non-vectorizable queries. The conditions of this scenario
> are as follows:
> # 1. The query {{SELECT}}s non-primitive type columns. I have tested this
> with {{ARRAY}} and {{RECORD}} types.
> # 2. The query involves {{JOIN}}ing of two or more tables.
> # 3. The tables are external, and they use the same storage type. This may
> occur with internal tables as well, but I am unable to test this in the Hive
> instance I have access to. The storage types I have tested this with are
> Parquet and Avro. If all tables use Parquet or all use Avro, the error is
> raised. If there is a mixture of storage types, the query runs successfully.
> I tried to dig through code to find and fix the issue myself, but
> unfortunately I did not have a complete enough understanding of Hive and its
> dependencies to pinpoint the issue.
> Further examples and explanation - given the following two table definitions:
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test (
> int_col BIGINT,
> array_col ARRAY ,
> record_col STRUCT
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col", "type": ["null",
> "long"]},
> {"name": "array_col",
> "type": ["null", {"items": ["null", "long"], "type": "array"}]},
> {"name": "record_col",
> "type": ["null",
> {"fields": [{"name": "a", "type": ["null", "long"]},
> {"name": "b", "type": ["null", "long"]}],
>"name": "record_col_0",
>"type": "record"}]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> {code:sql}
> CREATE EXTERNAL TABLE experimental.join_test2 (
> int_col2 BIGINT,
> str_col STRING
> )
> STORED AS AVRO
> LOCATION 's3://s3-bucket/join_test2/'
> TBLPROPERTIES (
> 'avro.schema.literal'='{"fields": [{"name": "int_col2", "type": ["null",
> "long"]},
> {"name": "str_col", "type": ["null", "string"]}],
> "name": "Root",
> "type": "record"}'
> )
> {code}
> I can successfully query both of these tables, running the following queries:
> {code:sql}
> SELECT * FROM experimental.join_test;
> 1 [1,2] {"a":1,"b":2}
> 2 [3,4] {"a":3,"b":4}
> 3 [5,6] {"a":5,"b":6}
> {code}
> {code:sql}
> SELECT * FROM experimental.join_test2;
> 1 should-appear-after-joining
> 2 should-appear-after-joining
> 3 should-appear-after-joining
> 4 shouldnt-appear-after-joining
> {code}
> I can also join the tables together, so long as I do not select the complex
> type columns:
> {code:sql}
> SELECT int_col, int_col2, str_col
> FROM experimental.join_test AS jt
> JOIN experimental.join_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> 3 3 should-appear-after-joining
> 2 2 should-appear-after-joining
> 1 1 should-appear-after-joining
> {code}
> But as soon as complex columns are introduced to the {{SELECT}} clause, an
> error arises.
> {code:sql}
> SELECT *
> FROM experimental.george_test AS jt
> JOIN experimental.george_test2 AS jt2 ON jt.int_col = jt2.int_col2;
> {code}
> I will attach the full stack trace as a file - it is rather large. The
> lowest-level error message given is slightly different depending on which
> complex type is involved in raising the error.
> * If it is an {{ARRAY}} type, the error message is
> {{org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector}}
> * * If it is a {{RECORD}} type, the error message is
>
[jira] [Updated] (HIVE-24647) Some Non-Vectorizable Queries are Run as Vectorized - Leads to Query Failures
[
https://issues.apache.org/jira/browse/HIVE-24647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Wood updated HIVE-24647:
---
Description:
A scenario has been discovered that seems to result in attempts to vectorize
and run non-vectorizable queries. The conditions of this scenario are as
follows:A scenario has been discovered that seems to result in attempts to
vectorize and run non-vectorizable queries. The conditions of this scenario are
as follows:
# 1. The query {{SELECT}}s non-primitive type columns. I have tested this with
{{ARRAY}} and {{RECORD}} types.
# 2. The query involves {{JOIN}}ing of two or more tables.
# 3. The tables are external, and they use the same storage type. This may
occur with internal tables as well, but I am unable to test this in the Hive
instance I have access to. The storage types I have tested this with are
Parquet and Avro. If all tables use Parquet or all use Avro, the error is
raised. If there is a mixture of storage types, the query runs successfully.
I tried to dig through code to find and fix the issue myself, but unfortunately
I did not have a complete enough understanding of Hive and its dependencies to
pinpoint the issue.
Further examples and explanation - given the following two table definitions:
{code:sql}
CREATE EXTERNAL TABLE experimental.join_test (
int_col BIGINT,
array_col ARRAY ,
record_col STRUCT
)
STORED AS AVRO
LOCATION 's3://s3-bucket/join_test/'
TBLPROPERTIES (
'avro.schema.literal'='{"fields": [{"name": "int_col", "type": ["null",
"long"]},
{"name": "array_col",
"type": ["null", {"items": ["null", "long"], "type": "array"}]},
{"name": "record_col",
"type": ["null",
{"fields": [{"name": "a", "type": ["null", "long"]},
{"name": "b", "type": ["null", "long"]}],
"name": "record_col_0",
"type": "record"}]}],
"name": "Root",
"type": "record"}'
)
{code}
{code:sql}
CREATE EXTERNAL TABLE experimental.join_test2 (
int_col2 BIGINT,
str_col STRING
)
STORED AS AVRO
LOCATION 's3://s3-bucket/join_test2/'
TBLPROPERTIES (
'avro.schema.literal'='{"fields": [{"name": "int_col2", "type": ["null",
"long"]},
{"name": "str_col", "type": ["null", "string"]}],
"name": "Root",
"type": "record"}'
)
{code}
I can successfully query both of these tables, running the following queries:
{code:sql}
SELECT * FROM experimental.join_test;
1 [1,2] {"a":1,"b":2}
2 [3,4] {"a":3,"b":4}
3 [5,6] {"a":5,"b":6}
{code}
{code:sql}
SELECT * FROM experimental.join_test2;
1 should-appear-after-joining
2 should-appear-after-joining
3 should-appear-after-joining
4 shouldnt-appear-after-joining
{code}
I can also join the tables together, so long as I do not select the complex
type columns:
{code:sql}
SELECT int_col, int_col2, str_col
FROM experimental.join_test AS jt
JOIN experimental.join_test2 AS jt2 ON jt.int_col = jt2.int_col2;
3 3 should-appear-after-joining
2 2 should-appear-after-joining
1 1 should-appear-after-joining
{code}
But as soon as complex columns are introduced to the {{SELECT}} clause, an
error arises.
{code:sql}
SELECT *
FROM experimental.george_test AS jt
JOIN experimental.george_test2 AS jt2 ON jt.int_col = jt2.int_col2;
{code}
I will attach the full stack trace as a file - it is rather large. The
lowest-level error message given is slightly different depending on which
complex type is involved in raising the error.
* If it is an {{ARRAY}} type, the error message is
{{org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector}}
* * If it is a {{RECORD}} type, the error message is
{{org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector}}
If the tables are of different types - for example, if the table {{join_test}}
was stored as Parquet instead of Avro - the query executes without issue.
>From what I can tell by looking at the stack trace, Hive is attempting to run
>this query as a vectorized query, leading to the error.Some further points of
>support for this idea:
# 1. Vectorization is not supported in queries that involved complex type
columns, which lines up with the fact that queries only fail when complex type
columns are included in the query.
# 2. Explicitly setting {{hive.vectorized.execution.enabled}} to {{'false'}}
allows these queries to execute and finish normally.
was:
A scenario has been discovered that seems to result in attempts to vectorize
and run non-vectorizable queries. The conditions of this scenario are as
follows:A scenario has been discovered that seems t
[jira] [Work logged] (HIVE-24534) Prevent comparisons between characters and decimals types when strict checks enabled
[
https://issues.apache.org/jira/browse/HIVE-24534?focusedWorklogId=536670&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536670
]
ASF GitHub Bot logged work on HIVE-24534:
-
Author: ASF GitHub Bot
Created on: 16/Jan/21 00:16
Start Date: 16/Jan/21 00:16
Worklog Time Spent: 10m
Work Description: zabetak commented on pull request #1780:
URL: https://github.com/apache/hive/pull/1780#issuecomment-761269576
Updated PR to improve readability
(https://github.com/apache/hive/pull/1780/commits/37e707705c6cc7c025bbe7fb1b67682062dc84c7)
and at the same time address HIVE-24646.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536670)
Time Spent: 1.5h (was: 1h 20m)
> Prevent comparisons between characters and decimals types when strict checks
> enabled
>
>
> Key: HIVE-24534
> URL: https://issues.apache.org/jira/browse/HIVE-24534
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When we compare decimal and character types implicit conversions take place
> that can lead to unexpected and surprising results.
> {code:sql}
> create table t_str (str_col string);
> insert into t_str values ('1208925742523269458163819');select * from t_str
> where str_col=1208925742523269479013976;
> {code}
> The SELECT query brings up one row while the filtering value is not the same
> with the one present in the string column of the table. The problem is that
> both types are converted to doubles and due to loss of precision the values
> are deemed equal.
> Even if we change the implicit conversion to use another type (HIVE-24528)
> there are always some cases that may lead to unexpected results.
> The goal of this issue is to prevent comparisons between decimal and
> character types when hive.strict.checks.type.safety is enabled and throw an
> error.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24603) ALTER TABLE RENAME is not modifying the location of managed table
[ https://issues.apache.org/jira/browse/HIVE-24603?focusedWorklogId=536665&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536665 ] ASF GitHub Bot logged work on HIVE-24603: - Author: ASF GitHub Bot Created on: 16/Jan/21 00:11 Start Date: 16/Jan/21 00:11 Worklog Time Spent: 10m Work Description: saihemanth-cloudera closed pull request #1841: URL: https://github.com/apache/hive/pull/1841 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536665) Time Spent: 20m (was: 10m) > ALTER TABLE RENAME is not modifying the location of managed table > - > > Key: HIVE-24603 > URL: https://issues.apache.org/jira/browse/HIVE-24603 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The location of the managed table is not changing when the table is renamed. > This causes correctness issues as well like the following - > create table abc (id int); > insert into abc values (1); > rename table abc to def; > create table abc (id int); // This should be empty > insert into abc values (2); > select * from abc ; // now returns the 1 and 2, (ie the old results as well) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24646) Strict type checks are not enforced between bigints and doubles
[
https://issues.apache.org/jira/browse/HIVE-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-24646:
--
> Strict type checks are not enforced between bigints and doubles
>
>
> Key: HIVE-24646
> URL: https://issues.apache.org/jira/browse/HIVE-24646
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> When the {{hive.strict.checks.type.safety}} property is set to true, queries
> with comparisons between bigints and doubles should fail according to the
> description of the property.
> At the moment a warning message is displayed in the console but the query
> doesn't fail no matter the value of the property.
> {noformat}
> WARNING: Comparing a bigint and a double may result in a loss of precision.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266421#comment-17266421
]
Stamatis Zampetakis commented on HIVE-24569:
Thanks [~prasanth_j], I replied in the PR. If you think that 5 sec of
inactivity is very low then we can increase this value; we may keep the
descriptor open for a bit longer but it is better if it is not on us to manage
the lifecycle of appenders.
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121804_ce53eeb5-c73f-4999-b7a4-b4dd04d4e4de-dag_1608659125567_0008_197.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 974 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224122002_1693bd7d-2f0e-4673-a8d1-b7cb14a02204-dag_1608659125567_0008_204.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 989 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121909_6a56218f-06c7-4906-9907-4b6dd824b100-dag_1608659125567_0008_201.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 984 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121754_78ef49a0-bc23-478f-9a16-87fa25e7a287-dag_1608659125567_0008_196.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 983 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121855_e65b9ebf-b2ec-4159-9570-1904442b7048-dag_1608659125567_0008_200.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 981 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121818_e9051ae3-1316-46af-aabb-22c53ed2fda7-dag_1608659125567_0008_198.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 980 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121744_fcf37921-4351-4368-95ee-b5be2592d
[jira] [Work logged] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?focusedWorklogId=536663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536663
]
ASF GitHub Bot logged work on HIVE-24569:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 23:41
Start Date: 15/Jan/21 23:41
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #1858:
URL: https://github.com/apache/hive/pull/1858#discussion_r558673024
##
File path: llap-server/src/main/resources/llap-daemon-log4j2.properties
##
@@ -81,23 +81,21 @@ appender.query-routing.type = Routing
appender.query-routing.name = query-routing
appender.query-routing.routes.type = Routes
appender.query-routing.routes.pattern = $${ctx:queryId}
-#Purge polciy for query-based Routing Appender
-appender.query-routing.purgePolicy.type = LlapRoutingAppenderPurgePolicy
-# Note: Do not change this name without changing the corresponding entry in
LlapConstants
-appender.query-routing.purgePolicy.name = llapLogPurgerQueryRouting
+#Purge policy for query-based Routing Appender
+appender.query-routing.purgePolicy.type = IdlePurgePolicy
+appender.query-routing.purgePolicy.timeToLive = 5
Review comment:
If 5 sec pass without a write then the appender is deleted and the file
is closed. The appender/file will reopen again when logging activity comes
back. Do you think this might be a problem?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536663)
Time Spent: 40m (was: 0.5h)
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121804_ce5
[jira] [Resolved] (HIVE-24603) ALTER TABLE RENAME is not modifying the location of managed table
[ https://issues.apache.org/jira/browse/HIVE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-24603. -- Resolution: Fixed > ALTER TABLE RENAME is not modifying the location of managed table > - > > Key: HIVE-24603 > URL: https://issues.apache.org/jira/browse/HIVE-24603 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The location of the managed table is not changing when the table is renamed. > This causes correctness issues as well like the following - > create table abc (id int); > insert into abc values (1); > rename table abc to def; > create table abc (id int); // This should be empty > insert into abc values (2); > select * from abc ; // now returns the 1 and 2, (ie the old results as well) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24603) ALTER TABLE RENAME is not modifying the location of managed table
[ https://issues.apache.org/jira/browse/HIVE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266372#comment-17266372 ] Naveen Gangam commented on HIVE-24603: -- This fix has been committed to master. Thank you for the contribution [~hemanth619]. > ALTER TABLE RENAME is not modifying the location of managed table > - > > Key: HIVE-24603 > URL: https://issues.apache.org/jira/browse/HIVE-24603 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The location of the managed table is not changing when the table is renamed. > This causes correctness issues as well like the following - > create table abc (id int); > insert into abc values (1); > rename table abc to def; > create table abc (id int); // This should be empty > insert into abc values (2); > select * from abc ; // now returns the 1 and 2, (ie the old results as well) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23553) Upgrade ORC version to 1.6.7
[ https://issues.apache.org/jira/browse/HIVE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated HIVE-23553: - Summary: Upgrade ORC version to 1.6.7 (was: Upgrade ORC version to 1.6.X) > Upgrade ORC version to 1.6.7 > > > Key: HIVE-23553 > URL: https://issues.apache.org/jira/browse/HIVE-23553 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Apache Hive is currently on 1.5.X version and in order to take advantage of > the latest ORC improvements such as column encryption we have to bump to > 1.6.X. > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343288&styleName=&projectId=12318320&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED_4ae78f19321c7fb1e7f337fba1dd90af751d8810_lin > Even though ORC reader could work out of the box, HIVE LLAP is heavily > depending on internal ORC APIs e.g., to retrieve and store File Footers, > Tails, streams – un/compress RG data etc. As there ware many internal changes > from 1.5 to 1.6 (Input stream offsets, relative BufferChunks etc.) the > upgrade is not straightforward. > This Umbrella Jira tracks this upgrade effort. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24470) Separate HiveMetastore Thrift and Driver logic
[ https://issues.apache.org/jira/browse/HIVE-24470?focusedWorklogId=536607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536607 ] ASF GitHub Bot logged work on HIVE-24470: - Author: ASF GitHub Bot Created on: 15/Jan/21 19:38 Start Date: 15/Jan/21 19:38 Worklog Time Spent: 10m Work Description: vihangk1 commented on a change in pull request #1787: URL: https://github.com/apache/hive/pull/1787#discussion_r558549035 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java ## @@ -0,0 +1,10155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.metastore; + +import com.codahale.metrics.Counter; +import com.codahale.metrics.Timer; +import com.facebook.fb303.FacebookBase; +import com.facebook.fb303.fb_status; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; +import com.google.common.base.Splitter; +import com.google.common.base.Supplier; +import com.google.common.base.Suppliers; +import com.google.common.collect.Lists; +import com.google.common.util.concurrent.ThreadFactoryBuilder; +import org.apache.commons.collections.CollectionUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.common.AcidConstants; +import org.apache.hadoop.hive.common.AcidMetaDataFile; +import org.apache.hadoop.hive.common.StatsSetupConst; +import org.apache.hadoop.hive.common.TableName; +import org.apache.hadoop.hive.common.ValidReaderWriteIdList; +import org.apache.hadoop.hive.common.ValidWriteIdList; +import org.apache.hadoop.hive.common.repl.ReplConst; +import org.apache.hadoop.hive.metastore.api.*; +import org.apache.hadoop.hive.metastore.conf.MetastoreConf; +import org.apache.hadoop.hive.metastore.events.AbortTxnEvent; +import org.apache.hadoop.hive.metastore.events.AcidWriteEvent; +import org.apache.hadoop.hive.metastore.events.AddCheckConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddDefaultConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddForeignKeyEvent; +import org.apache.hadoop.hive.metastore.events.AddNotNullConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddPartitionEvent; +import org.apache.hadoop.hive.metastore.events.AddPrimaryKeyEvent; +import org.apache.hadoop.hive.metastore.events.AddSchemaVersionEvent; +import org.apache.hadoop.hive.metastore.events.AddUniqueConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AllocWriteIdEvent; +import org.apache.hadoop.hive.metastore.events.AlterCatalogEvent; +import org.apache.hadoop.hive.metastore.events.AlterDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.AlterISchemaEvent; +import org.apache.hadoop.hive.metastore.events.AlterPartitionEvent; +import org.apache.hadoop.hive.metastore.events.AlterSchemaVersionEvent; +import org.apache.hadoop.hive.metastore.events.AlterTableEvent; +import org.apache.hadoop.hive.metastore.events.CommitTxnEvent; +import org.apache.hadoop.hive.metastore.events.ConfigChangeEvent; +import org.apache.hadoop.hive.metastore.events.CreateCatalogEvent; +import org.apache.hadoop.hive.metastore.events.CreateDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.CreateFunctionEvent; +import org.apache.hadoop.hive.metastore.events.CreateISchemaEvent; +import org.apache.hadoop.hive.metastore.events.CreateTableEvent; +import org.apache.hadoop.hive.metastore.events.DeletePartitionColumnStatEvent; +import org.apache.hadoop.hive.metastore.events.DeleteTableColumnStatEvent; +import org.apache.hadoop.hive.metastore.events.DropCatalogEvent; +import org.apache.hadoop.hive.metastore.events.DropConstraintEvent; +import org.apache.hadoop.hive.metastore.events.DropDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.DropFunctionEvent; +import org.apache.hadoop.hive.metastore.events.DropISchemaEvent; +import org.apache.hadoop.hive.metastore.events.DropPartitionEvent; +import org.apache.h
[jira] [Work logged] (HIVE-24470) Separate HiveMetastore Thrift and Driver logic
[ https://issues.apache.org/jira/browse/HIVE-24470?focusedWorklogId=536606&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536606 ] ASF GitHub Bot logged work on HIVE-24470: - Author: ASF GitHub Bot Created on: 15/Jan/21 19:37 Start Date: 15/Jan/21 19:37 Worklog Time Spent: 10m Work Description: vihangk1 commented on a change in pull request #1787: URL: https://github.com/apache/hive/pull/1787#discussion_r558548201 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java ## @@ -0,0 +1,10155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hive.metastore; + +import com.codahale.metrics.Counter; +import com.codahale.metrics.Timer; +import com.facebook.fb303.FacebookBase; +import com.facebook.fb303.fb_status; +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Preconditions; +import com.google.common.base.Splitter; +import com.google.common.base.Supplier; +import com.google.common.base.Suppliers; +import com.google.common.collect.Lists; +import com.google.common.util.concurrent.ThreadFactoryBuilder; +import org.apache.commons.collections.CollectionUtils; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.common.AcidConstants; +import org.apache.hadoop.hive.common.AcidMetaDataFile; +import org.apache.hadoop.hive.common.StatsSetupConst; +import org.apache.hadoop.hive.common.TableName; +import org.apache.hadoop.hive.common.ValidReaderWriteIdList; +import org.apache.hadoop.hive.common.ValidWriteIdList; +import org.apache.hadoop.hive.common.repl.ReplConst; +import org.apache.hadoop.hive.metastore.api.*; +import org.apache.hadoop.hive.metastore.conf.MetastoreConf; +import org.apache.hadoop.hive.metastore.events.AbortTxnEvent; +import org.apache.hadoop.hive.metastore.events.AcidWriteEvent; +import org.apache.hadoop.hive.metastore.events.AddCheckConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddDefaultConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddForeignKeyEvent; +import org.apache.hadoop.hive.metastore.events.AddNotNullConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AddPartitionEvent; +import org.apache.hadoop.hive.metastore.events.AddPrimaryKeyEvent; +import org.apache.hadoop.hive.metastore.events.AddSchemaVersionEvent; +import org.apache.hadoop.hive.metastore.events.AddUniqueConstraintEvent; +import org.apache.hadoop.hive.metastore.events.AllocWriteIdEvent; +import org.apache.hadoop.hive.metastore.events.AlterCatalogEvent; +import org.apache.hadoop.hive.metastore.events.AlterDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.AlterISchemaEvent; +import org.apache.hadoop.hive.metastore.events.AlterPartitionEvent; +import org.apache.hadoop.hive.metastore.events.AlterSchemaVersionEvent; +import org.apache.hadoop.hive.metastore.events.AlterTableEvent; +import org.apache.hadoop.hive.metastore.events.CommitTxnEvent; +import org.apache.hadoop.hive.metastore.events.ConfigChangeEvent; +import org.apache.hadoop.hive.metastore.events.CreateCatalogEvent; +import org.apache.hadoop.hive.metastore.events.CreateDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.CreateFunctionEvent; +import org.apache.hadoop.hive.metastore.events.CreateISchemaEvent; +import org.apache.hadoop.hive.metastore.events.CreateTableEvent; +import org.apache.hadoop.hive.metastore.events.DeletePartitionColumnStatEvent; +import org.apache.hadoop.hive.metastore.events.DeleteTableColumnStatEvent; +import org.apache.hadoop.hive.metastore.events.DropCatalogEvent; +import org.apache.hadoop.hive.metastore.events.DropConstraintEvent; +import org.apache.hadoop.hive.metastore.events.DropDatabaseEvent; +import org.apache.hadoop.hive.metastore.events.DropFunctionEvent; +import org.apache.hadoop.hive.metastore.events.DropISchemaEvent; +import org.apache.hadoop.hive.metastore.events.DropPartitionEvent; +import org.apache.h
[jira] [Commented] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266271#comment-17266271
]
Prasanth Jayachandran commented on HIVE-24569:
--
Left a question about how the Idle purging is triggered (we wanted to avoid
files being closed too frequently). Looks good otherwise, +1. Thanks for adding
tests for it!
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121804_ce53eeb5-c73f-4999-b7a4-b4dd04d4e4de-dag_1608659125567_0008_197.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 974 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224122002_1693bd7d-2f0e-4673-a8d1-b7cb14a02204-dag_1608659125567_0008_204.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 989 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121909_6a56218f-06c7-4906-9907-4b6dd824b100-dag_1608659125567_0008_201.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 984 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121754_78ef49a0-bc23-478f-9a16-87fa25e7a287-dag_1608659125567_0008_196.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 983 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121855_e65b9ebf-b2ec-4159-9570-1904442b7048-dag_1608659125567_0008_200.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 981 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121818_e9051ae3-1316-46af-aabb-22c53ed2fda7-dag_1608659125567_0008_198.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 980 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121744_fcf37921-4351-4368-95ee-b5be2592d89a-dag_1608659125567_0008_195.log
> lrwx-- 1 hive hadoop 64 Dec 24 12
[jira] [Work logged] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?focusedWorklogId=536574&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536574
]
ASF GitHub Bot logged work on HIVE-24569:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 18:09
Start Date: 15/Jan/21 18:09
Worklog Time Spent: 10m
Work Description: prasanthj commented on a change in pull request #1858:
URL: https://github.com/apache/hive/pull/1858#discussion_r558482661
##
File path: llap-server/src/main/resources/llap-daemon-log4j2.properties
##
@@ -81,23 +81,21 @@ appender.query-routing.type = Routing
appender.query-routing.name = query-routing
appender.query-routing.routes.type = Routes
appender.query-routing.routes.pattern = $${ctx:queryId}
-#Purge polciy for query-based Routing Appender
-appender.query-routing.purgePolicy.type = LlapRoutingAppenderPurgePolicy
-# Note: Do not change this name without changing the corresponding entry in
LlapConstants
-appender.query-routing.purgePolicy.name = llapLogPurgerQueryRouting
+#Purge policy for query-based Routing Appender
+appender.query-routing.purgePolicy.type = IdlePurgePolicy
+appender.query-routing.purgePolicy.timeToLive = 5
Review comment:
Question: When does this timer start? after the log file is closed or
after every writes? If "idle" is triggered after closing the file I think it is
ok. But if it is triggered after the last write then 5 second is probably very
low as even if llap is stuck in gc pause it could trigger a file close.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536574)
Time Spent: 0.5h (was: 20m)
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
>
[jira] [Comment Edited] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266250#comment-17266250
]
Jesus Camacho Rodriguez edited comment on HIVE-24569 at 1/15/21, 5:59 PM:
--
[~prasanth_j], [~mustafaiman], could any of you review this patch? Thanks
was (Author: jcamachorodriguez):
[~prasanthj], [~mustafaiman], could any of you review this patch? Thanks
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121804_ce53eeb5-c73f-4999-b7a4-b4dd04d4e4de-dag_1608659125567_0008_197.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 974 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224122002_1693bd7d-2f0e-4673-a8d1-b7cb14a02204-dag_1608659125567_0008_204.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 989 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121909_6a56218f-06c7-4906-9907-4b6dd824b100-dag_1608659125567_0008_201.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 984 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121754_78ef49a0-bc23-478f-9a16-87fa25e7a287-dag_1608659125567_0008_196.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 983 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121855_e65b9ebf-b2ec-4159-9570-1904442b7048-dag_1608659125567_0008_200.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 981 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121818_e9051ae3-1316-46af-aabb-22c53ed2fda7-dag_1608659125567_0008_198.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 980 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121744_fcf37921-4351-4368-95ee-b5be2592d89a-dag_160
[jira] [Commented] (HIVE-24569) LLAP daemon leaks file descriptors/log4j appenders
[
https://issues.apache.org/jira/browse/HIVE-24569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266250#comment-17266250
]
Jesus Camacho Rodriguez commented on HIVE-24569:
[~prasanthj], [~mustafaiman], could any of you review this patch? Thanks
> LLAP daemon leaks file descriptors/log4j appenders
> --
>
> Key: HIVE-24569
> URL: https://issues.apache.org/jira/browse/HIVE-24569
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: llap-appender-gc-roots.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With HIVE-9756 query logs in LLAP are directed to different files (file per
> query) using a Log4j2 routing appender. Without a purge policy in place,
> appenders are created dynamically by the routing appender, one for each
> query, and remain in memory forever. The dynamic appenders write to files so
> each appender holds to a file descriptor.
> Further work HIVE-14224 has mitigated the issue by introducing a custom
> purging policy (LlapRoutingAppenderPurgePolicy) which deletes the dynamic
> appenders (and closes the respective files) when the query is completed
> (org.apache.hadoop.hive.llap.daemon.impl.QueryTracker#handleLogOnQueryCompletion).
>
> However, in the presence of multiple threads appending to the logs there are
> race conditions. In an internal Hive cluster the number of file descriptors
> started going up approx one descriptor leaking per query. After some
> debugging it turns out that one thread (running the
> QueryTracker#handleLogOnQueryCompletion) signals that the query has finished
> and thus the purge policy should get rid of the respective appender (and
> close the file) while another (Task-Executor-0) attempts to append another
> log message for the same query. The initial appender is closed after the
> request from the query tracker but a new one is created to accomodate the
> message from the task executor and the latter is never removed thus creating
> a leak.
> Similar leaks have been identified and fixed for HS2 with the most similar
> one being that described
> [here|https://issues.apache.org/jira/browse/HIVE-22753?focusedCommentId=17021041&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17021041].
>
> The problem relies on the timing of threads so it may not manifestate in all
> versions between 2.2.0 and 4.0.0. Usually the leak can be seen either via
> lsof (or other similar command) with the following output:
> {noformat}
> # 1494391 is the PID of the LLAP daemon process
> ls -ltr /proc/1494391/fd
> ...
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 978 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121724_66ce273d-54a9-4dcd-a9fb-20cb5691cef7-dag_1608659125567_0008_194.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 977 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121804_ce53eeb5-c73f-4999-b7a4-b4dd04d4e4de-dag_1608659125567_0008_197.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 974 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224122002_1693bd7d-2f0e-4673-a8d1-b7cb14a02204-dag_1608659125567_0008_204.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 989 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121909_6a56218f-06c7-4906-9907-4b6dd824b100-dag_1608659125567_0008_201.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 984 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121754_78ef49a0-bc23-478f-9a16-87fa25e7a287-dag_1608659125567_0008_196.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 983 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121855_e65b9ebf-b2ec-4159-9570-1904442b7048-dag_1608659125567_0008_200.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 981 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121818_e9051ae3-1316-46af-aabb-22c53ed2fda7-dag_1608659125567_0008_198.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 980 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01_02/hive_20201224121744_fcf37921-4351-4368-95ee-b5be2592d89a-dag_1608659125567_0008_195.log
> lrwx-- 1 hive hadoop 64 Dec 24 12:08 979 ->
> /hadoop/yarn/log/application_1608659125567_0006/container_e04_1608659125567_0006_01
[jira] [Updated] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24645: -- Labels: pull-request-available (was: ) > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?focusedWorklogId=536563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536563 ] ASF GitHub Bot logged work on HIVE-24645: - Author: ASF GitHub Bot Created on: 15/Jan/21 17:47 Start Date: 15/Jan/21 17:47 Worklog Time Spent: 10m Work Description: jfsii opened a new pull request #1876: URL: https://github.com/apache/hive/pull/1876 Change-Id: Ic6ddbf9ffb14c293caadf1f8ec1e7ec0f9291a31 ### What changes were proposed in this pull request? UDF configure does not get called if fetch task conversion happens. This attempts to resolve this by calling configure with a dummy MapredContext. It is a bit ugly in that MapredContext seems to be legacy baggage with various members that do not make sense for a variety of engines and code. ### Why are the changes needed? Some custom UDFs are designed with the expectation that configure is called in all instances. Users sometimes use it to inspect configuration values or to set new values to propagate to the evaluate method or to do one time setup. ### Does this PR introduce _any_ user-facing change? Yes - it will now call configure in cases where configure was not called before. ### How was this patch tested? Added udf_configure.q and manual testing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536563) Remaining Estimate: 0h Time Spent: 10m > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24601) Control CBO fallback behavior via property
[
https://issues.apache.org/jira/browse/HIVE-24601?focusedWorklogId=536559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536559
]
ASF GitHub Bot logged work on HIVE-24601:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 17:26
Start Date: 15/Jan/21 17:26
Worklog Time Spent: 10m
Work Description: zabetak opened a new pull request #1875:
URL: https://github.com/apache/hive/pull/1875
### What changes were proposed in this pull request?
1. Add `hive.cbo.fallback.strategy` and plug it to the planner
2. Replace usage of `hive.in.test` in planner with
`hive.cbo.fallback.strategy`
### Why are the changes needed?
1. Provide the means to fail-fast when CBO failures appear (even in
production)
2. Allow finer control on tests requiring to fail on CBO, legacy, or both
### Does this PR introduce _any_ user-facing change?
No at its current state.
I was thinking to even drop the `CONSERVATIVE` option. The CBO error is
always logged (not hidden) so I don't see why we should complicate the decision
more.
### How was this patch tested?
If we agree on the approach, I will add some tests for all options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536559)
Remaining Estimate: 0h
Time Spent: 10m
> Control CBO fallback behavior via property
> --
>
> Key: HIVE-24601
> URL: https://issues.apache.org/jira/browse/HIVE-24601
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When CBO optimizer fails there is a fallback mechanism(HIVE-7413) that will
> retry to process the query using the legacy Hive optimizer.
> There are use-cases where this behavior is not desirable notably for the
> tests (HIVE-16058) but also for end users who would like to disable the
> fall-back mechanism to avoid running problematic queries without realizing.
> The goal of this issue is to introduce a dedicated Hive property controlling
> this behavior,{{hive.cbo.fallback.enable}}, for both tests and production.
> The default value should be true and tests should run with this property set
> to false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24601) Control CBO fallback behavior via property
[
https://issues.apache.org/jira/browse/HIVE-24601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24601:
--
Labels: pull-request-available (was: )
> Control CBO fallback behavior via property
> --
>
> Key: HIVE-24601
> URL: https://issues.apache.org/jira/browse/HIVE-24601
> Project: Hive
> Issue Type: Improvement
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When CBO optimizer fails there is a fallback mechanism(HIVE-7413) that will
> retry to process the query using the legacy Hive optimizer.
> There are use-cases where this behavior is not desirable notably for the
> tests (HIVE-16058) but also for end users who would like to disable the
> fall-back mechanism to avoid running problematic queries without realizing.
> The goal of this issue is to introduce a dedicated Hive property controlling
> this behavior,{{hive.cbo.fallback.enable}}, for both tests and production.
> The default value should be true and tests should run with this property set
> to false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24428) Concurrent add_partitions requests may lead to data loss
[
https://issues.apache.org/jira/browse/HIVE-24428?focusedWorklogId=536553&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536553
]
ASF GitHub Bot logged work on HIVE-24428:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 17:07
Start Date: 15/Jan/21 17:07
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1724:
URL: https://github.com/apache/hive/pull/1724#discussion_r558441940
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -281,6 +279,7 @@ public static boolean isRenameAllowed(Database srcDB,
Database destDB) {
static AtomicInteger databaseCount, tableCount, partCount;
private Warehouse wh; // hdfs warehouse
+private static Striped tablelocks;
Review comment:
oh, sorry, i missed that synchronization is done on class level not
instance.
Note: there is a bug in synchronization, double checked locking should be
used here or if check (threadPool == null) should be moved inside the critical
section.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536553)
Time Spent: 5h 50m (was: 5h 40m)
> Concurrent add_partitions requests may lead to data loss
>
>
> Key: HIVE-24428
> URL: https://issues.apache.org/jira/browse/HIVE-24428
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> in case multiple clients are adding partitions to the same table - when the
> same partition is being added there is a chance that the data dir is removed
> after the other client have already written its data
> https://github.com/apache/hive/blob/5e96b14a2357c66a0640254d5414bc706d8be852/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3958
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24428) Concurrent add_partitions requests may lead to data loss
[
https://issues.apache.org/jira/browse/HIVE-24428?focusedWorklogId=536552&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536552
]
ASF GitHub Bot logged work on HIVE-24428:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 17:04
Start Date: 15/Jan/21 17:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #1724:
URL: https://github.com/apache/hive/pull/1724#discussion_r558441940
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##
@@ -281,6 +279,7 @@ public static boolean isRenameAllowed(Database srcDB,
Database destDB) {
static AtomicInteger databaseCount, tableCount, partCount;
private Warehouse wh; // hdfs warehouse
+private static Striped tablelocks;
Review comment:
oh, sorry, i missed that synchronization is done on class level not
instance.
Note: there is a bug in synchronization, double checked locking should be
used here or if check should be moved inside the critical section.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536552)
Time Spent: 5h 40m (was: 5.5h)
> Concurrent add_partitions requests may lead to data loss
>
>
> Key: HIVE-24428
> URL: https://issues.apache.org/jira/browse/HIVE-24428
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> in case multiple clients are adding partitions to the same table - when the
> same partition is being added there is a chance that the data dir is removed
> after the other client have already written its data
> https://github.com/apache/hive/blob/5e96b14a2357c66a0640254d5414bc706d8be852/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L3958
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24626) LLAP: reader threads could be starvated if all IO elevator threads are busy to enqueue to another readers with full queue
[
https://issues.apache.org/jira/browse/HIVE-24626?focusedWorklogId=536551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536551
]
ASF GitHub Bot logged work on HIVE-24626:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 16:58
Start Date: 15/Jan/21 16:58
Worklog Time Spent: 10m
Work Description: abstractdog edited a comment on pull request #1868:
URL: https://github.com/apache/hive/pull/1868#issuecomment-760918210
thanks for the review @prasanthj, @pgaref ! I'm trying to achieve a green
test and then I'll merge this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536551)
Time Spent: 0.5h (was: 20m)
> LLAP: reader threads could be starvated if all IO elevator threads are busy
> to enqueue to another readers with full queue
> -
>
> Key: HIVE-24626
> URL: https://issues.apache.org/jira/browse/HIVE-24626
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: executor_stack_cache_none_12_io_threads.log
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The root cause is that the readers cannot queue.offer items to full queues,
> which belong to consumers that are blocked on other consumers.
> Scenario is like below:
> {code}
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 2 .. llap RUNNING 3 210
> 0 0
> Map 1 llap RUNNING676 0 119 557
> 0 0
> Map 3 llap RUNNING108 0 21 87
> 0 21
> Reducer 4 llapINITED 1 001
> 0 0
> Map 5 llapINITED108 00 108
> 0 0
> Reducer 6 llapINITED 4 004
> 0 0
> Reducer 7 llapINITED 1 001
> 0 0
> --
> VERTICES: 00/07 [>>--] 0%ELAPSED TIME: 3489.83 s
> --
> {code}
> Map2 is MAPJOINed to Map1. In an LLAP daemon, the forever running Map2 task
> is blocked on nextCvb:
> {code}
> "TezTR-886270_0_1_0_1_0" #154 daemon prio=5 os_prio=0 tid=0x7f1b88348000
> nid=0x147 waiting on condition [0x7f0ce005d000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f0de8025e00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.nextCvb(LlapRecordReader.java:517)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:372)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:82)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:362)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:117)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:115)
> at
> org.apache.hadoop.hiv
[jira] [Assigned] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-24645: --- > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24515) Analyze table job can be skipped when stats populated are already accurate
[
https://issues.apache.org/jira/browse/HIVE-24515?focusedWorklogId=536541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536541
]
ASF GitHub Bot logged work on HIVE-24515:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 15:52
Start Date: 15/Jan/21 15:52
Worklog Time Spent: 10m
Work Description: maheshk114 commented on a change in pull request #1834:
URL: https://github.com/apache/hive/pull/1834#discussion_r558397066
##
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java
##
@@ -204,6 +206,54 @@ public int persistColumnStats(Hive db, Table tbl) throws
HiveException, MetaExce
public void setDpPartSpecs(Collection dpPartSpecs) {
}
+ public static boolean canSkipStatsGeneration(String dbName, String tblName,
String partName,
+ long statsWriteId, String
queryValidWriteIdList) {
+if (queryValidWriteIdList != null) { // Can be null if its not an ACID
table.
+ ValidWriteIdList validWriteIdList = new
ValidReaderWriteIdList(queryValidWriteIdList);
+ // Just check if the write ID is valid. If it's valid (i.e. we are
allowed to see it),
+ // that means it cannot possibly be a concurrent write. As stats
optimization is enabled
+ // only in case auto gather is enabled. Thus the stats must be updated
by a valid committed
+ // transaction and stats generation can be skipped.
+ if (validWriteIdList.isWriteIdValid(statsWriteId)) {
+try {
+ IMetaStoreClient msc = Hive.get().getMSC();
+ TxnState state = msc.findStatStatusByWriteId(dbName, tblName,
partName, statsWriteId);
Review comment:
There is 2 cases where we need to compute stats again
1. If the table is updated by a txn with auto gather stats set to false. In
that case, we can find a txn with id greater than the stats write id (txn) in
the completed txn table. That is done in findStatStatusByWriteId method.
2. If the table stats is updated by an aborted txn. That can be. checked
using valid write id list. But if the compaction has cleaned up the txn info,
we can not judge that. So we have to check the completed txn table.That is done
in findStatStatusByWriteId method.
If the stat is updated by a read only txn (that is stats updater), then also
we need not compute the stats again. The info about read only txns is not
present in the valid write id list. That is done in findStatStatusByWriteId
method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536541)
Time Spent: 1h 10m (was: 1h)
> Analyze table job can be skipped when stats populated are already accurate
> --
>
> Key: HIVE-24515
> URL: https://issues.apache.org/jira/browse/HIVE-24515
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> For non-partitioned tables, stats detail should be present in table level,
> e.g
> {noformat}
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"d_current_day":"true"...
> }}
> {noformat}
> For partitioned tables, stats detail should be present in partition level,
> {noformat}
> store_sales(ss_sold_date_sk=2451819)
> {totalSize=0, numRows=0, rawDataSize=0,
> COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"ss_addr_sk":"true"}}
>
> {noformat}
> When stats populated are already accurate, {{analyze table tn compute
> statistics for columns}} should skip launching the job.
>
> For ACID tables, stats are auto computed and it can skip computing stats
> again when stats are accurate.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24644) QueryResultCache parses the query twice
[ https://issues.apache.org/jira/browse/HIVE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24644: -- Labels: pull-request-available (was: ) > QueryResultCache parses the query twice > --- > > Key: HIVE-24644 > URL: https://issues.apache.org/jira/browse/HIVE-24644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Query result cache lookup results by query text which has fully resolved > table references. > In order to generate this query text currently implementation > * transforms the AST tree back to String > * parses the String generated in above step > * traverse the new AST and replaces the table references to the fully > qualified ones > * transforms the new AST tree back to String -> this will be the cache key -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24644) QueryResultCache parses the query twice
[ https://issues.apache.org/jira/browse/HIVE-24644?focusedWorklogId=536535&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536535 ] ASF GitHub Bot logged work on HIVE-24644: - Author: ASF GitHub Bot Created on: 15/Jan/21 15:05 Start Date: 15/Jan/21 15:05 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request #1874: URL: https://github.com/apache/hive/pull/1874 ### What changes were proposed in this pull request? Query results cache requires query text having fully qualified table names as cache key. By the time query compilation reach the point where results cache key is generated unparseTranslator instance has the fully qualified table names. Use this to generate cache key. Generating the key from query text also requires the TokenRewriteStream instance related to the parsed query. Applying transformations stored in the unparseTranslator would alter the TokenRewriteStream and makes invalid for further usage. In order to avoid this a dedicated TokenRewriteStream program is introduced for Query results cache. ### Why are the changes needed? All query was parsed twice: * first parse to have the AST tree for compilation * second parse to generate cache key from query text having fully qualified table names. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Run existing qtest about Results cache: ``` mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=results_cache_invalidation2.q,results_cache_with_masking.q,results_cache_lifetime.q,results_cache_temptable.q,results_cache_with_auth.q,results_cache_3.q,results_cache_1.q,results_cache_empty_result.q,results_cache_capacity.q,results_cache_diff_fs.q,results_cache_2.q,results_cache_truncate.q,results_cache_quoted_identifiers.q,results_cache_transactional.q,results_cache_invalidation.q -pl itests/qtest -Pitests ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536535) Remaining Estimate: 0h Time Spent: 10m > QueryResultCache parses the query twice > --- > > Key: HIVE-24644 > URL: https://issues.apache.org/jira/browse/HIVE-24644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Query result cache lookup results by query text which has fully resolved > table references. > In order to generate this query text currently implementation > * transforms the AST tree back to String > * parses the String generated in above step > * traverse the new AST and replaces the table references to the fully > qualified ones > * transforms the new AST tree back to String -> this will be the cache key -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24644) QueryResultCache parses the query twice
[ https://issues.apache.org/jira/browse/HIVE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-24644: -- Priority: Minor (was: Major) > QueryResultCache parses the query twice > --- > > Key: HIVE-24644 > URL: https://issues.apache.org/jira/browse/HIVE-24644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > > Query result cache lookup results by query text which has fully resolved > table references. > In order to generate this query text currently implementation > * transforms the AST tree back to String > * parses the String generated in above step > * traverse the new AST and replaces the table references to the fully > qualified ones > * transforms the new AST tree back to String -> this will be the cache key -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24644) QueryResultCache parses the query twice
[ https://issues.apache.org/jira/browse/HIVE-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-24644: - > QueryResultCache parses the query twice > --- > > Key: HIVE-24644 > URL: https://issues.apache.org/jira/browse/HIVE-24644 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, Parser >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > > Query result cache lookup results by query text which has fully resolved > table references. > In order to generate this query text currently implementation > * transforms the AST tree back to String > * parses the String generated in above step > * traverse the new AST and replaces the table references to the fully > qualified ones > * transforms the new AST tree back to String -> this will be the cache key -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24606) Multi-stage materialized CTEs can lose intermediate data
[
https://issues.apache.org/jira/browse/HIVE-24606?focusedWorklogId=536513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536513
]
ASF GitHub Bot logged work on HIVE-24606:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 14:23
Start Date: 15/Jan/21 14:23
Worklog Time Spent: 10m
Work Description: okumin commented on a change in pull request #1873:
URL: https://github.com/apache/hive/pull/1873#discussion_r558338464
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -1411,47 +1414,92 @@ public String toString() {
}
}
- private List> toRealRootTasks(List execution) {
-List> cteRoots = new ArrayList<>();
-List> cteLeafs = new ArrayList<>();
-List> curTopRoots = null;
-List> curBottomLeafs = null;
-for (CTEClause current : execution) {
- if (current.parents.isEmpty() && curTopRoots != null) {
-cteRoots.addAll(curTopRoots);
-cteLeafs.addAll(curBottomLeafs);
-curTopRoots = curBottomLeafs = null;
Review comment:
This looks like the root cause. In [the case I put in the
ticket](https://issues.apache.org/jira/browse/HIVE-24606), there are the
following dependencies.
- ` -> a1`
- ` -> x`
- ` -> a2`
- `a2 -> a1`
But the old implementation tries to traverse CTEs in order of `a1` -> `x` ->
`a2` -> ``, the order which depends on AST.
As a result, when it visits `a2`, the information of `a1` has gone and it
fails to link `a2` with `a1`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536513)
Time Spent: 0.5h (was: 20m)
> Multi-stage materialized CTEs can lose intermediate data
>
>
> Key: HIVE-24606
> URL: https://issues.apache.org/jira/browse/HIVE-24606
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.3.7, 3.1.2, 4.0.0
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> With complex multi-stage CTEs, Hive can start a latter stage before its
> previous stage finishes.
> That's because `SemanticAnalyzer#toRealRootTasks` can fail to resolve
> dependency between multistage materialized CTEs when a non-materialized CTE
> cuts in.
>
> [https://github.com/apache/hive/blob/425e1ff7c054f87c4db87e77d004282d529599ae/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1414]
>
> For example, when submitting this query,
> {code:sql}
> SET hive.optimize.cte.materialize.threshold=2;
> SET hive.optimize.cte.materialize.full.aggregate.only=false;
> WITH x AS ( SELECT 'x' AS id ), -- not materialized
> a1 AS ( SELECT 'a1' AS id ), -- materialized by a2 and the root
> a2 AS ( SELECT 'a2 <- ' || id AS id FROM a1) -- materialized by the root
> SELECT * FROM a1
> UNION ALL
> SELECT * FROM x
> UNION ALL
> SELECT * FROM a2
> UNION ALL
> SELECT * FROM a2;
> {code}
> `toRealRootTask` will traverse the CTEs in order of `a1`, `x`, and `a2`. It
> means the dependency between `a1` and `a2` will be ignored and `a2` can start
> without waiting for `a1`. As a result, the above query returns the following
> result.
> {code:java}
> +-+
> | id |
> +-+
> | a1 |
> | x |
> +-+
> {code}
> For your information, I ran this test with revision =
> 425e1ff7c054f87c4db87e77d004282d529599ae.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24606) Multi-stage materialized CTEs can lose intermediate data
[
https://issues.apache.org/jira/browse/HIVE-24606?focusedWorklogId=536492&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536492
]
ASF GitHub Bot logged work on HIVE-24606:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 13:58
Start Date: 15/Jan/21 13:58
Worklog Time Spent: 10m
Work Description: okumin commented on a change in pull request #1873:
URL: https://github.com/apache/hive/pull/1873#discussion_r558323016
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -1343,7 +1344,9 @@ private void addCTEAsSubQuery(QB qb, String cteName,
String cteAlias)
@Override
public List> getAllRootTasks() {
if (!rootTasksResolved) {
- rootTasks = toRealRootTasks(rootClause.asExecutionOrder());
+ List execution = rootClause.asExecutionOrder();
+ linkRealDependencies(execution);
+ rootTasks = toRealRootTasks(execution);
Review comment:
Separated mutating tasks and picking up real roots to avoid complexity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536492)
Time Spent: 20m (was: 10m)
> Multi-stage materialized CTEs can lose intermediate data
>
>
> Key: HIVE-24606
> URL: https://issues.apache.org/jira/browse/HIVE-24606
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.3.7, 3.1.2, 4.0.0
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> With complex multi-stage CTEs, Hive can start a latter stage before its
> previous stage finishes.
> That's because `SemanticAnalyzer#toRealRootTasks` can fail to resolve
> dependency between multistage materialized CTEs when a non-materialized CTE
> cuts in.
>
> [https://github.com/apache/hive/blob/425e1ff7c054f87c4db87e77d004282d529599ae/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1414]
>
> For example, when submitting this query,
> {code:sql}
> SET hive.optimize.cte.materialize.threshold=2;
> SET hive.optimize.cte.materialize.full.aggregate.only=false;
> WITH x AS ( SELECT 'x' AS id ), -- not materialized
> a1 AS ( SELECT 'a1' AS id ), -- materialized by a2 and the root
> a2 AS ( SELECT 'a2 <- ' || id AS id FROM a1) -- materialized by the root
> SELECT * FROM a1
> UNION ALL
> SELECT * FROM x
> UNION ALL
> SELECT * FROM a2
> UNION ALL
> SELECT * FROM a2;
> {code}
> `toRealRootTask` will traverse the CTEs in order of `a1`, `x`, and `a2`. It
> means the dependency between `a1` and `a2` will be ignored and `a2` can start
> without waiting for `a1`. As a result, the above query returns the following
> result.
> {code:java}
> +-+
> | id |
> +-+
> | a1 |
> | x |
> +-+
> {code}
> For your information, I ran this test with revision =
> 425e1ff7c054f87c4db87e77d004282d529599ae.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24514) UpdateMDatabaseURI does not update managed location URI
[ https://issues.apache.org/jira/browse/HIVE-24514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-24514. -- Resolution: Fixed Thanks for the review! Merged to master. > UpdateMDatabaseURI does not update managed location URI > --- > > Key: HIVE-24514 > URL: https://issues.apache.org/jira/browse/HIVE-24514 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > When FS Root is updated using metatool, if the DB has managed location > defined, > updateMDatabaseURI API should update the managed location as well. Currently > it only updates location uri. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24514) UpdateMDatabaseURI does not update managed location URI
[ https://issues.apache.org/jira/browse/HIVE-24514?focusedWorklogId=536487&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536487 ] ASF GitHub Bot logged work on HIVE-24514: - Author: ASF GitHub Bot Created on: 15/Jan/21 13:52 Start Date: 15/Jan/21 13:52 Worklog Time Spent: 10m Work Description: prasanthj merged pull request #1761: URL: https://github.com/apache/hive/pull/1761 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536487) Time Spent: 50m (was: 40m) > UpdateMDatabaseURI does not update managed location URI > --- > > Key: HIVE-24514 > URL: https://issues.apache.org/jira/browse/HIVE-24514 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > When FS Root is updated using metatool, if the DB has managed location > defined, > updateMDatabaseURI API should update the managed location as well. Currently > it only updates location uri. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24606) Multi-stage materialized CTEs can lose intermediate data
[
https://issues.apache.org/jira/browse/HIVE-24606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24606:
--
Labels: pull-request-available (was: )
> Multi-stage materialized CTEs can lose intermediate data
>
>
> Key: HIVE-24606
> URL: https://issues.apache.org/jira/browse/HIVE-24606
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.3.7, 3.1.2, 4.0.0
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With complex multi-stage CTEs, Hive can start a latter stage before its
> previous stage finishes.
> That's because `SemanticAnalyzer#toRealRootTasks` can fail to resolve
> dependency between multistage materialized CTEs when a non-materialized CTE
> cuts in.
>
> [https://github.com/apache/hive/blob/425e1ff7c054f87c4db87e77d004282d529599ae/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1414]
>
> For example, when submitting this query,
> {code:sql}
> SET hive.optimize.cte.materialize.threshold=2;
> SET hive.optimize.cte.materialize.full.aggregate.only=false;
> WITH x AS ( SELECT 'x' AS id ), -- not materialized
> a1 AS ( SELECT 'a1' AS id ), -- materialized by a2 and the root
> a2 AS ( SELECT 'a2 <- ' || id AS id FROM a1) -- materialized by the root
> SELECT * FROM a1
> UNION ALL
> SELECT * FROM x
> UNION ALL
> SELECT * FROM a2
> UNION ALL
> SELECT * FROM a2;
> {code}
> `toRealRootTask` will traverse the CTEs in order of `a1`, `x`, and `a2`. It
> means the dependency between `a1` and `a2` will be ignored and `a2` can start
> without waiting for `a1`. As a result, the above query returns the following
> result.
> {code:java}
> +-+
> | id |
> +-+
> | a1 |
> | x |
> +-+
> {code}
> For your information, I ran this test with revision =
> 425e1ff7c054f87c4db87e77d004282d529599ae.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24606) Multi-stage materialized CTEs can lose intermediate data
[
https://issues.apache.org/jira/browse/HIVE-24606?focusedWorklogId=536485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536485
]
ASF GitHub Bot logged work on HIVE-24606:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 13:46
Start Date: 15/Jan/21 13:46
Worklog Time Spent: 10m
Work Description: okumin opened a new pull request #1873:
URL: https://github.com/apache/hive/pull/1873
### What changes were proposed in this pull request?
Build correct dependencies among CTEs in order to prevent wrong results.
### Why are the changes needed?
A Hive query with `hive.optimize.cte.materialize.threshold` can return wrong
results when it has complex CTEs.
The issue can happen when multistage CTEs have dependencies and
SemanticAnalyzer fails to link their tasks.
https://issues.apache.org/jira/browse/HIVE-24606
### Does this PR introduce _any_ user-facing change?
No. Just a bug fix.
### How was this patch tested?
This PR adds three test cases. All fail when using the latest
revision(`58552a0c6b42988efb5160b045a3bf985477f117`).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536485)
Remaining Estimate: 0h
Time Spent: 10m
> Multi-stage materialized CTEs can lose intermediate data
>
>
> Key: HIVE-24606
> URL: https://issues.apache.org/jira/browse/HIVE-24606
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.3.7, 3.1.2, 4.0.0
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With complex multi-stage CTEs, Hive can start a latter stage before its
> previous stage finishes.
> That's because `SemanticAnalyzer#toRealRootTasks` can fail to resolve
> dependency between multistage materialized CTEs when a non-materialized CTE
> cuts in.
>
> [https://github.com/apache/hive/blob/425e1ff7c054f87c4db87e77d004282d529599ae/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1414]
>
> For example, when submitting this query,
> {code:sql}
> SET hive.optimize.cte.materialize.threshold=2;
> SET hive.optimize.cte.materialize.full.aggregate.only=false;
> WITH x AS ( SELECT 'x' AS id ), -- not materialized
> a1 AS ( SELECT 'a1' AS id ), -- materialized by a2 and the root
> a2 AS ( SELECT 'a2 <- ' || id AS id FROM a1) -- materialized by the root
> SELECT * FROM a1
> UNION ALL
> SELECT * FROM x
> UNION ALL
> SELECT * FROM a2
> UNION ALL
> SELECT * FROM a2;
> {code}
> `toRealRootTask` will traverse the CTEs in order of `a1`, `x`, and `a2`. It
> means the dependency between `a1` and `a2` will be ignored and `a2` can start
> without waiting for `a1`. As a result, the above query returns the following
> result.
> {code:java}
> +-+
> | id |
> +-+
> | a1 |
> | x |
> +-+
> {code}
> For your information, I ran this test with revision =
> 425e1ff7c054f87c4db87e77d004282d529599ae.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24278) Implement an UDF for throwing exception in arbitrary vertex
[
https://issues.apache.org/jira/browse/HIVE-24278?focusedWorklogId=536480&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536480
]
ASF GitHub Bot logged work on HIVE-24278:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 13:26
Start Date: 15/Jan/21 13:26
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1817:
URL: https://github.com/apache/hive/pull/1817#discussion_r558304711
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFExceptionInVertex.java
##
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.udf.generic;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.hadoop.hive.ql.exec.Description;
+import org.apache.hadoop.hive.ql.exec.MapredContext;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.exec.tez.TezProcessor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableConstantIntObjectInspector;
+import
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableConstantStringObjectInspector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements the UDF which can throw an exception in arbitrary
vertex (typically mapper)
+ * / task / task attempt. For throwing exception in reducer side, where most
probably
+ * GroupByOperator codepath applies, GenericUDAFExceptionInVertex is used.
+ */
+@Description(name = "exception_in_vertex_udf", value = "_FUNC_(vertexName,
taskNumberExpression, taskAttemptNumberExpression)")
+public class GenericUDFExceptionInVertex extends GenericUDF {
+ private static final Logger LOG =
LoggerFactory.getLogger(GenericUDFExceptionInVertex.class);
+
+ private String vertexName;
+ private String taskNumberExpr;
+ private String taskAttemptNumberExpr;
+ private String currentVertexName;
+ private int currentTaskNumber;
+ private int currentTaskAttemptNumber;
+ private boolean alreadyCheckedAndPassed;
+
+ @Override
+ public ObjectInspector initialize(ObjectInspector[] parameters) throws
UDFArgumentException {
+if (parameters.length < 2) {
+ throw new UDFArgumentTypeException(-1,
+ "At least two argument is expected (fake column ref, vertex name)");
+}
+
+this.vertexName = getVertexName(parameters, 1);
+this.taskNumberExpr = getTaskNumber(parameters, 2);
+this.taskAttemptNumberExpr = getTaskAttemptNumber(parameters, 3);
Review comment:
sure, fixed in new commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536480)
Time Spent: 50m (was: 40m)
> Implement an UDF for throwing exception in arbitrary vertex
> ---
>
> Key: HIVE-24278
> URL: https://issues.apache.org/jira/browse/HIVE-24278
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> For testing purposes sometimes we need to make the query fail in a vertex, so
> assuming that we already know the plan, it could be something like:
> on mapper side
> {code}
> select a.col1, exception_in_vertex("Map 1") from a
> join b on b.id = a.id
> {code}
> or on reducer side
> {code
[jira] [Assigned] (HIVE-24632) Replace with null when GenericUDFBaseCompare has a non-interpretable val
[
https://issues.apache.org/jira/browse/HIVE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng reassigned HIVE-24632:
--
Assignee: Zhihua Deng
> Replace with null when GenericUDFBaseCompare has a non-interpretable val
>
>
> Key: HIVE-24632
> URL: https://issues.apache.org/jira/browse/HIVE-24632
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The query
> {code:java}
> create table ccn_table(key int, value string);
> set hive.cbo.enable=false;
> select * from ccn_table where key > '123a' ;
> {code}
> will scan all records(partitions) compared to older version, as the plan
> tells:
> {noformat}
> STAGE PLANS:
> Stage: Stage-0
>Fetch Operator
> limit: -1
> Processor Tree:
>TableScan
> alias: ccn_table
> filterExpr: (key > '123a') (type: boolean)
> Statistics: Num rows: 2 Data size: 180 Basic stats: COMPLETE Column
> stats: COMPLETE
> GatherStats: false
> Filter Operator
>isSamplingPred: false
>predicate: (key > '123a') (type: boolean)
>Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE Column
> stats: COMPLETE
>Select Operator
> expressions: key (type: int), value (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink{noformat}
> When the TypeCheckProcFactory#getXpathOrFuncExprNodeDesc validates the expr:
> +key > '123a',+ the operator(>) is not an equal operator(=), so the factory
> returns +key > '123a'+ as it is. However all the subclass of
> GenericUDFBaseCompare(except GenericUDFOPEqualNS and GenericUDFOPNotEqualNS)
> would return null if either side of the function children is null, so it's
> safe to return constant null when processing the expr +`key > '123a'`+. This
> will benifit some queries when the cbo is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24632) Replace with null when GenericUDFBaseCompare has a non-interpretable val
[
https://issues.apache.org/jira/browse/HIVE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266037#comment-17266037
]
Zhihua Deng commented on HIVE-24632:
Thanks a lot for the merge and review, [~kgyrtkirk]!
> Replace with null when GenericUDFBaseCompare has a non-interpretable val
>
>
> Key: HIVE-24632
> URL: https://issues.apache.org/jira/browse/HIVE-24632
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The query
> {code:java}
> create table ccn_table(key int, value string);
> set hive.cbo.enable=false;
> select * from ccn_table where key > '123a' ;
> {code}
> will scan all records(partitions) compared to older version, as the plan
> tells:
> {noformat}
> STAGE PLANS:
> Stage: Stage-0
>Fetch Operator
> limit: -1
> Processor Tree:
>TableScan
> alias: ccn_table
> filterExpr: (key > '123a') (type: boolean)
> Statistics: Num rows: 2 Data size: 180 Basic stats: COMPLETE Column
> stats: COMPLETE
> GatherStats: false
> Filter Operator
>isSamplingPred: false
>predicate: (key > '123a') (type: boolean)
>Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE Column
> stats: COMPLETE
>Select Operator
> expressions: key (type: int), value (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink{noformat}
> When the TypeCheckProcFactory#getXpathOrFuncExprNodeDesc validates the expr:
> +key > '123a',+ the operator(>) is not an equal operator(=), so the factory
> returns +key > '123a'+ as it is. However all the subclass of
> GenericUDFBaseCompare(except GenericUDFOPEqualNS and GenericUDFOPNotEqualNS)
> would return null if either side of the function children is null, so it's
> safe to return constant null when processing the expr +`key > '123a'`+. This
> will benifit some queries when the cbo is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24626) LLAP: reader threads could be starvated if all IO elevator threads are busy to enqueue to another readers with full queue
[
https://issues.apache.org/jira/browse/HIVE-24626?focusedWorklogId=536453&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536453
]
ASF GitHub Bot logged work on HIVE-24626:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 12:38
Start Date: 15/Jan/21 12:38
Worklog Time Spent: 10m
Work Description: abstractdog commented on pull request #1868:
URL: https://github.com/apache/hive/pull/1868#issuecomment-760918210
thanks for the review @prasanthj ! I'm trying to achieve a green test and
then I'll merge this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536453)
Time Spent: 20m (was: 10m)
> LLAP: reader threads could be starvated if all IO elevator threads are busy
> to enqueue to another readers with full queue
> -
>
> Key: HIVE-24626
> URL: https://issues.apache.org/jira/browse/HIVE-24626
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: executor_stack_cache_none_12_io_threads.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The root cause is that the readers cannot queue.offer items to full queues,
> which belong to consumers that are blocked on other consumers.
> Scenario is like below:
> {code}
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 2 .. llap RUNNING 3 210
> 0 0
> Map 1 llap RUNNING676 0 119 557
> 0 0
> Map 3 llap RUNNING108 0 21 87
> 0 21
> Reducer 4 llapINITED 1 001
> 0 0
> Map 5 llapINITED108 00 108
> 0 0
> Reducer 6 llapINITED 4 004
> 0 0
> Reducer 7 llapINITED 1 001
> 0 0
> --
> VERTICES: 00/07 [>>--] 0%ELAPSED TIME: 3489.83 s
> --
> {code}
> Map2 is MAPJOINed to Map1. In an LLAP daemon, the forever running Map2 task
> is blocked on nextCvb:
> {code}
> "TezTR-886270_0_1_0_1_0" #154 daemon prio=5 os_prio=0 tid=0x7f1b88348000
> nid=0x147 waiting on condition [0x7f0ce005d000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f0de8025e00> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.nextCvb(LlapRecordReader.java:517)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:372)
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.next(LlapRecordReader.java:82)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:362)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:117)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:115)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRe
[jira] [Work logged] (HIVE-24629) Invoke optional output committer in TezProcessor
[ https://issues.apache.org/jira/browse/HIVE-24629?focusedWorklogId=536445&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536445 ] ASF GitHub Bot logged work on HIVE-24629: - Author: ASF GitHub Bot Created on: 15/Jan/21 11:54 Start Date: 15/Jan/21 11:54 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #1857: URL: https://github.com/apache/hive/pull/1857#issuecomment-760898737 LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536445) Time Spent: 40m (was: 0.5h) > Invoke optional output committer in TezProcessor > > > Key: HIVE-24629 > URL: https://issues.apache.org/jira/browse/HIVE-24629 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > In order to enable Hive to write to Iceberg tables, we need to use an output > committer which will fire at the end of each Tez task execution (commitTask) > and the after the execution of each vertex (commitOutput/commitJob). This > output committer will issue a commit containing the written-out data files to > the Iceberg table, replacing its previous snapshot pointer with a new one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24629) Invoke optional output committer in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-24629?focusedWorklogId=536439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536439
]
ASF GitHub Bot logged work on HIVE-24629:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 11:37
Start Date: 15/Jan/21 11:37
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1857:
URL: https://github.com/apache/hive/pull/1857#discussion_r558250464
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##
@@ -1600,9 +1601,14 @@ public Vertex createVertex(JobConf conf, BaseWork
workUnit, Path scratchDir,
// final vertices need to have at least one output
boolean endVertex = tezWork.getLeaves().contains(workUnit);
if (endVertex) {
+ OutputCommitterDescriptor ocd = null;
+ if (conf.get("hive.tez.mapreduce.output.committer.class") != null
+ && conf.get("mapred.output.committer.class") != null) {
Review comment:
thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536439)
Time Spent: 0.5h (was: 20m)
> Invoke optional output committer in TezProcessor
>
>
> Key: HIVE-24629
> URL: https://issues.apache.org/jira/browse/HIVE-24629
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In order to enable Hive to write to Iceberg tables, we need to use an output
> committer which will fire at the end of each Tez task execution (commitTask)
> and the after the execution of each vertex (commitOutput/commitJob). This
> output committer will issue a commit containing the written-out data files to
> the Iceberg table, replacing its previous snapshot pointer with a new one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24629) Invoke optional output committer in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-24629?focusedWorklogId=536428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536428
]
ASF GitHub Bot logged work on HIVE-24629:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 11:07
Start Date: 15/Jan/21 11:07
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #1857:
URL: https://github.com/apache/hive/pull/1857#discussion_r558236568
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##
@@ -1600,9 +1601,14 @@ public Vertex createVertex(JobConf conf, BaseWork
workUnit, Path scratchDir,
// final vertices need to have at least one output
boolean endVertex = tezWork.getLeaves().contains(workUnit);
if (endVertex) {
+ OutputCommitterDescriptor ocd = null;
+ if (conf.get("hive.tez.mapreduce.output.committer.class") != null
+ && conf.get("mapred.output.committer.class") != null) {
Review comment:
After thinking about it, we shouldn't have this check. Just because
`MROutputCommitter` delegates all the calls to an inner committer object
(implementation of this inner field is determined by
`mapred.output.committer.class`), it doesn't mean all Tez output committer
implementations should or will in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536428)
Time Spent: 20m (was: 10m)
> Invoke optional output committer in TezProcessor
>
>
> Key: HIVE-24629
> URL: https://issues.apache.org/jira/browse/HIVE-24629
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In order to enable Hive to write to Iceberg tables, we need to use an output
> committer which will fire at the end of each Tez task execution (commitTask)
> and the after the execution of each vertex (commitOutput/commitJob). This
> output committer will issue a commit containing the written-out data files to
> the Iceberg table, replacing its previous snapshot pointer with a new one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24643) Access Operation state directly where possible
[ https://issues.apache.org/jira/browse/HIVE-24643?focusedWorklogId=536427&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536427 ] ASF GitHub Bot logged work on HIVE-24643: - Author: ASF GitHub Bot Created on: 15/Jan/21 11:02 Start Date: 15/Jan/21 11:02 Worklog Time Spent: 10m Work Description: pvargacl opened a new pull request #1872: URL: https://github.com/apache/hive/pull/1872 ### What changes were proposed in this pull request? Small refactor to void unnecessary JSON serialisation ### Why are the changes needed? Minor performance improvement ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? current unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536427) Remaining Estimate: 0h Time Spent: 10m > Access Operation state directly where possible > -- > > Key: HIVE-24643 > URL: https://issues.apache.org/jira/browse/HIVE-24643 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Operation state is accessed during query execution by calling > operation.getStatus, that will serialise TaskStatuses into Json, which can we > skipped if only the state is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24643) Access Operation state directly where possible
[ https://issues.apache.org/jira/browse/HIVE-24643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24643: -- Labels: pull-request-available (was: ) > Access Operation state directly where possible > -- > > Key: HIVE-24643 > URL: https://issues.apache.org/jira/browse/HIVE-24643 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Operation state is accessed during query execution by calling > operation.getStatus, that will serialise TaskStatuses into Json, which can we > skipped if only the state is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24643) Access Operation state directly where possible
[ https://issues.apache.org/jira/browse/HIVE-24643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga reassigned HIVE-24643: -- > Access Operation state directly where possible > -- > > Key: HIVE-24643 > URL: https://issues.apache.org/jira/browse/HIVE-24643 > Project: Hive > Issue Type: Improvement >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > > Operation state is accessed during query execution by calling > operation.getStatus, that will serialise TaskStatuses into Json, which can we > skipped if only the state is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24632) Replace with null when GenericUDFBaseCompare has a non-interpretable val
[
https://issues.apache.org/jira/browse/HIVE-24632?focusedWorklogId=536409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536409
]
ASF GitHub Bot logged work on HIVE-24632:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 10:38
Start Date: 15/Jan/21 10:38
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #1863:
URL: https://github.com/apache/hive/pull/1863
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536409)
Time Spent: 50m (was: 40m)
> Replace with null when GenericUDFBaseCompare has a non-interpretable val
>
>
> Key: HIVE-24632
> URL: https://issues.apache.org/jira/browse/HIVE-24632
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The query
> {code:java}
> create table ccn_table(key int, value string);
> set hive.cbo.enable=false;
> select * from ccn_table where key > '123a' ;
> {code}
> will scan all records(partitions) compared to older version, as the plan
> tells:
> {noformat}
> STAGE PLANS:
> Stage: Stage-0
>Fetch Operator
> limit: -1
> Processor Tree:
>TableScan
> alias: ccn_table
> filterExpr: (key > '123a') (type: boolean)
> Statistics: Num rows: 2 Data size: 180 Basic stats: COMPLETE Column
> stats: COMPLETE
> GatherStats: false
> Filter Operator
>isSamplingPred: false
>predicate: (key > '123a') (type: boolean)
>Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE Column
> stats: COMPLETE
>Select Operator
> expressions: key (type: int), value (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink{noformat}
> When the TypeCheckProcFactory#getXpathOrFuncExprNodeDesc validates the expr:
> +key > '123a',+ the operator(>) is not an equal operator(=), so the factory
> returns +key > '123a'+ as it is. However all the subclass of
> GenericUDFBaseCompare(except GenericUDFOPEqualNS and GenericUDFOPNotEqualNS)
> would return null if either side of the function children is null, so it's
> safe to return constant null when processing the expr +`key > '123a'`+. This
> will benifit some queries when the cbo is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-24632) Replace with null when GenericUDFBaseCompare has a non-interpretable val
[
https://issues.apache.org/jira/browse/HIVE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-24632.
-
Fix Version/s: 4.0.0
Resolution: Fixed
merged into master. Thank you [~dengzh]!
> Replace with null when GenericUDFBaseCompare has a non-interpretable val
>
>
> Key: HIVE-24632
> URL: https://issues.apache.org/jira/browse/HIVE-24632
> Project: Hive
> Issue Type: Improvement
> Components: Parser
>Affects Versions: 4.0.0
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The query
> {code:java}
> create table ccn_table(key int, value string);
> set hive.cbo.enable=false;
> select * from ccn_table where key > '123a' ;
> {code}
> will scan all records(partitions) compared to older version, as the plan
> tells:
> {noformat}
> STAGE PLANS:
> Stage: Stage-0
>Fetch Operator
> limit: -1
> Processor Tree:
>TableScan
> alias: ccn_table
> filterExpr: (key > '123a') (type: boolean)
> Statistics: Num rows: 2 Data size: 180 Basic stats: COMPLETE Column
> stats: COMPLETE
> GatherStats: false
> Filter Operator
>isSamplingPred: false
>predicate: (key > '123a') (type: boolean)
>Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE Column
> stats: COMPLETE
>Select Operator
> expressions: key (type: int), value (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 1 Data size: 90 Basic stats: COMPLETE
> Column stats: COMPLETE
> ListSink{noformat}
> When the TypeCheckProcFactory#getXpathOrFuncExprNodeDesc validates the expr:
> +key > '123a',+ the operator(>) is not an equal operator(=), so the factory
> returns +key > '123a'+ as it is. However all the subclass of
> GenericUDFBaseCompare(except GenericUDFOPEqualNS and GenericUDFOPNotEqualNS)
> would return null if either side of the function children is null, so it's
> safe to return constant null when processing the expr +`key > '123a'`+. This
> will benifit some queries when the cbo is disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24642) Multiple file listing calls are executed in the MoveTask in case of direct inserts
[ https://issues.apache.org/jira/browse/HIVE-24642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora reassigned HIVE-24642: > Multiple file listing calls are executed in the MoveTask in case of direct > inserts > -- > > Key: HIVE-24642 > URL: https://issues.apache.org/jira/browse/HIVE-24642 > Project: Hive > Issue Type: Improvement >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > When inserting data into a table with dynamic partitioning with direct insert > on, the MoveTask performs several file listings to look up the newly created > partitions and files. Check if all files listings are necessary or it can be > optimized to do less listings. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24642) Multiple file listing calls are executed in the MoveTask in case of direct inserts
[ https://issues.apache.org/jira/browse/HIVE-24642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-24642: - Affects Version/s: 4.0.0 > Multiple file listing calls are executed in the MoveTask in case of direct > inserts > -- > > Key: HIVE-24642 > URL: https://issues.apache.org/jira/browse/HIVE-24642 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > When inserting data into a table with dynamic partitioning with direct insert > on, the MoveTask performs several file listings to look up the newly created > partitions and files. Check if all files listings are necessary or it can be > optimized to do less listings. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24629) Invoke optional output committer in TezProcessor
[
https://issues.apache.org/jira/browse/HIVE-24629?focusedWorklogId=536405&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536405
]
ASF GitHub Bot logged work on HIVE-24629:
-
Author: ASF GitHub Bot
Created on: 15/Jan/21 10:21
Start Date: 15/Jan/21 10:21
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1857:
URL: https://github.com/apache/hive/pull/1857#discussion_r558200073
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
##
@@ -1600,9 +1601,14 @@ public Vertex createVertex(JobConf conf, BaseWork
workUnit, Path scratchDir,
// final vertices need to have at least one output
boolean endVertex = tezWork.getLeaves().contains(workUnit);
if (endVertex) {
+ OutputCommitterDescriptor ocd = null;
+ if (conf.get("hive.tez.mapreduce.output.committer.class") != null
+ && conf.get("mapred.output.committer.class") != null) {
Review comment:
why is the presence of "mapred.output.committer.class" is needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 536405)
Remaining Estimate: 0h
Time Spent: 10m
> Invoke optional output committer in TezProcessor
>
>
> Key: HIVE-24629
> URL: https://issues.apache.org/jira/browse/HIVE-24629
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In order to enable Hive to write to Iceberg tables, we need to use an output
> committer which will fire at the end of each Tez task execution (commitTask)
> and the after the execution of each vertex (commitOutput/commitJob). This
> output committer will issue a commit containing the written-out data files to
> the Iceberg table, replacing its previous snapshot pointer with a new one.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24629) Invoke optional output committer in TezProcessor
[ https://issues.apache.org/jira/browse/HIVE-24629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24629: -- Labels: pull-request-available (was: ) > Invoke optional output committer in TezProcessor > > > Key: HIVE-24629 > URL: https://issues.apache.org/jira/browse/HIVE-24629 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In order to enable Hive to write to Iceberg tables, we need to use an output > committer which will fire at the end of each Tez task execution (commitTask) > and the after the execution of each vertex (commitOutput/commitJob). This > output committer will issue a commit containing the written-out data files to > the Iceberg table, replacing its previous snapshot pointer with a new one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24513) Advance write Id during AlterTableDropConstraint DDL
[ https://issues.apache.org/jira/browse/HIVE-24513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das resolved HIVE-24513. --- Resolution: Fixed > Advance write Id during AlterTableDropConstraint DDL > > > Key: HIVE-24513 > URL: https://issues.apache.org/jira/browse/HIVE-24513 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > For AlterTableDropConstraint related DDL tasks, although we might be > advancing the write ID, looks like it's not updated correctly during the > Analyzer phase. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23728) Run metastore verification tests during precommit
[ https://issues.apache.org/jira/browse/HIVE-23728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265848#comment-17265848 ] Zoltan Haindrich edited comment on HIVE-23728 at 1/15/21, 9:14 AM: --- if we could run these then we might be able to run those as well - IIRC there is still some issue with the derby tests; and we don't have a good oracle image candidate... I forgot to drag this patch further - will try to get it in during the next couple days ; after this is available you could probably experiment with ITestDbTxnManager as well was (Author: kgyrtkirk): if we could run these then we might be able to run those as well - IIRC there is still some issue with the derby tests; and we don't have a good oracle image candidate... > Run metastore verification tests during precommit > - > > Key: HIVE-23728 > URL: https://issues.apache.org/jira/browse/HIVE-23728 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23728) Run metastore verification tests during precommit
[ https://issues.apache.org/jira/browse/HIVE-23728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17265848#comment-17265848 ] Zoltan Haindrich commented on HIVE-23728: - if we could run these then we might be able to run those as well - IIRC there is still some issue with the derby tests; and we don't have a good oracle image candidate... > Run metastore verification tests during precommit > - > > Key: HIVE-23728 > URL: https://issues.apache.org/jira/browse/HIVE-23728 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23728) Run metastore verification tests during precommit
[ https://issues.apache.org/jira/browse/HIVE-23728?focusedWorklogId=536391&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-536391 ] ASF GitHub Bot logged work on HIVE-23728: - Author: ASF GitHub Bot Created on: 15/Jan/21 09:10 Start Date: 15/Jan/21 09:10 Worklog Time Spent: 10m Work Description: kgyrtkirk opened a new pull request #1154: URL: https://github.com/apache/hive/pull/1154 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 536391) Time Spent: 1h 20m (was: 1h 10m) > Run metastore verification tests during precommit > - > > Key: HIVE-23728 > URL: https://issues.apache.org/jira/browse/HIVE-23728 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
