[jira] [Work logged] (HIVE-24752) Returned operation's drilldown link may be broken
[
https://issues.apache.org/jira/browse/HIVE-24752?focusedWorklogId=555038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555038
]
ASF GitHub Bot logged work on HIVE-24752:
-
Author: ASF GitHub Bot
Created on: 20/Feb/21 03:06
Start Date: 20/Feb/21 03:06
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #1958:
URL: https://github.com/apache/hive/pull/1958#issuecomment-782546764
Hi @pvary @marton-bod any comments?
Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555038)
Time Spent: 20m (was: 10m)
> Returned operation's drilldown link may be broken
> -
>
> Key: HIVE-24752
> URL: https://issues.apache.org/jira/browse/HIVE-24752
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Since HIVE-23625, the path spec for the query page has changed from
> _query_page_ to _query_page.html_,
> {code:java}
> webServer.addServlet("query_page", "/query_page.html",
> QueryProfileServlet.class);{code}
> the drilldown link of the operation returned may be broken if
> hive.server2.show.operation.drilldown.link is enabled...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24310) Allow specified number of deserialize errors to be ignored
[ https://issues.apache.org/jira/browse/HIVE-24310?focusedWorklogId=555037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555037 ] ASF GitHub Bot logged work on HIVE-24310: - Author: ASF GitHub Bot Created on: 20/Feb/21 03:04 Start Date: 20/Feb/21 03:04 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #1607: URL: https://github.com/apache/hive/pull/1607#issuecomment-782546435 Hi @abstractdog @mmccline, any thoughts or comments here? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 555037) Time Spent: 2h 10m (was: 2h) > Allow specified number of deserialize errors to be ignored > -- > > Key: HIVE-24310 > URL: https://issues.apache.org/jira/browse/HIVE-24310 > Project: Hive > Issue Type: Improvement > Components: Operators >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > Sometimes we see some corrupted records in user's raw data, like one > corrupted in a file which contains over thousands of records, user has to > either give up all records or replay the whole data in order to run > successfully on hive, we should provide a way to ignore such corrupted > records. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287446#comment-17287446
]
Yuming Wang edited comment on HIVE-24797 at 2/20/21, 1:14 AM:
--
[~iemejia] Could you help verify other incompatible changes:
[https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20resolution%20in%20(Fixed)%20AND%20cf%5B12310191%5D%20%3D%20%22Incompatible%20change%22%20AND%20fixVersion%20in%20(1.9.0%2C%201.9.1%2C%201.10.0%2C%201.10.1)%20%20%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC]
was (Author: q79969786):
[~SteveNiemitz] Could you help verify other incompatible changes:
[https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20resolution%20in%20(Fixed)%20AND%20cf%5B12310191%5D%20%3D%20%22Incompatible%20change%22%20AND%20fixVersion%20in%20(1.9.0%2C%201.9.1%2C%201.10.0%2C%201.10.1)%20%20%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC]
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287446#comment-17287446
]
Yuming Wang commented on HIVE-24797:
[~SteveNiemitz] Could you help verify other incompatible changes:
[https://issues.apache.org/jira/issues/?jql=project%20%3D%20AVRO%20AND%20resolution%20in%20(Fixed)%20AND%20cf%5B12310191%5D%20%3D%20%22Incompatible%20change%22%20AND%20fixVersion%20in%20(1.9.0%2C%201.9.1%2C%201.10.0%2C%201.10.1)%20%20%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC]
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=555010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555010
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 20/Feb/21 00:55
Start Date: 20/Feb/21 00:55
Worklog Time Spent: 10m
Work Description: wangyum edited a comment on pull request #1994:
URL: https://github.com/apache/hive/pull/1994#issuecomment-781979028
cc @sunchao @iemejia @viirya
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555010)
Time Spent: 50m (was: 40m)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated HIVE-24797:
---
Description:
It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
{code:json}
{
"type": "record",
"name": "EventData",
"doc": "event data",
"fields": [
{"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
"string"}, "default": null }
]
}
{code}
{noformat}
org.apache.avro.AvroTypeException: Invalid default for field
ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
at org.apache.avro.Schema.validateDefault(Schema.java:1571)
at org.apache.avro.Schema.access$500(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:544)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
{noformat}
was:
It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
{code:json}
{
"type": "record",
"name": "EventData",
"doc": "event data",
"fields": [
{"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
"string"}, "default": null }
]
}
{code}
{noformat}
org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS: null
not a {"type":"array","items":"string"}
at org.apache.avro.Schema.validateDefault(Schema.java:1571)
at org.apache.avro.Schema.access$500(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:544)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
{noformat}
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field
> ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
>
[jira] [Work logged] (HIVE-24528) Wrong implicit type conversion when comparing decimals and strings
[
https://issues.apache.org/jira/browse/HIVE-24528?focusedWorklogId=555007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555007
]
ASF GitHub Bot logged work on HIVE-24528:
-
Author: ASF GitHub Bot
Created on: 20/Feb/21 00:46
Start Date: 20/Feb/21 00:46
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1771:
URL: https://github.com/apache/hive/pull/1771
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555007)
Time Spent: 1h 20m (was: 1h 10m)
> Wrong implicit type conversion when comparing decimals and strings
> --
>
> Key: HIVE-24528
> URL: https://issues.apache.org/jira/browse/HIVE-24528
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.0
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In many cases when comparing decimals and strings (literals/columns) the
> comparison is done using doubles which can create some quite unexpected
> results in the answers of queries.
> {code:sql}
> create table t_str (str_col string);
> insert into t_str values ('1208925742523269458163819');
> select * from t_str where str_col=1208925742523269479013976;
> {code}
> The SELECT query brings up one row while the filtering value is not the same
> with the one present in the string column of the table. The problem is that
> both types are converted to doubles and due to loss of precision the values
> are deemed equal.
> The same happens during the join of a decimal with a string type.
> {code:sql}
> create table t_dec (dec_col decimal(25,0));
> insert into t_dec values (1208925742523269479013976);
> select * from t_dec inner join t_str on dec_col=str_col;
> {code}
> The join result contains one row although the values are not equal.
> Implicit type conversions are working differently in every DBMS and for some
> of them (e.g., mysql) the above behavior is normal or not allowed at all
> (e.g. Postgres).
> In the past, Hive used to compare decimal with string columns by converting
> to decimals but this behavior changed in 2.3.0 (with HIVE-13380). It seems
> that this behavior change was not intentional since following jiras (e.g.,
> HIVE-18434) imply that comparison of decimals with strings should be done
> using decimals. Since decimal is an exact numeric it appears a more
> appropriate type for comparing decimals and strings.
> The goal of this issue is to change the implicit conversion of decimals with
> strings to doubles and use decimals instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24545) jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/HIVE-24545?focusedWorklogId=555008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555008
]
ASF GitHub Bot logged work on HIVE-24545:
-
Author: ASF GitHub Bot
Created on: 20/Feb/21 00:46
Start Date: 20/Feb/21 00:46
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #1789:
URL: https://github.com/apache/hive/pull/1789#issuecomment-782482366
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555008)
Time Spent: 20m (was: 10m)
> jdbc.HiveStatement: Number of rows is greater than Integer.MAX_VALUE
>
>
> Key: HIVE-24545
> URL: https://issues.apache.org/jira/browse/HIVE-24545
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I found this while IOW on TPCDS 10TB:
> {code}
> --
> VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> --
> Map 1 .. llap SUCCEEDED 4210 421000
> 0 362
> Reducer 2 .. llap SUCCEEDED10110100
> 0 2
> Reducer 3 .. llap SUCCEEDED 1009 100900
> 0 1
> --
> VERTICES: 03/03 [==>>] 100% ELAPSED TIME: 12613.62 s
> --
> 20/12/16 01:37:36 [main]: WARN jdbc.HiveStatement: Number of rows is greater
> than Integer.MAX_VALUE
> {code}
> my scenario was:
> {code}
> set hive.exec.max.dynamic.partitions=2000;
> drop table if exists test_sales_2;
> create table test_sales_2 like
> tpcds_bin_partitioned_acid_orc_1.store_sales;
> insert overwrite table test_sales_2 select * from
> tpcds_bin_partitioned_acid_orc_1.store_sales where ss_sold_date_sk >
> 2451868;
> {code}
> regarding affected row numbers:
> {code}
> select count(*) from tpcds_bin_partitioned_acid_orc_1.store_sales where
> ss_sold_date_sk > 2451868;
> +--+
> | _c0 |
> +--+
> | 12287871907 |
> +--+
> {code}
> I guess we should switch to long
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=554984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554984
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 23:55
Start Date: 19/Feb/21 23:55
Worklog Time Spent: 10m
Work Description: wangyum opened a new pull request #1994:
URL: https://github.com/apache/hive/pull/1994
### What changes were proposed in this pull request?
This pr disable validate default values when parsing Avro schemas.
### Why are the changes needed?
It will throw exception when upgrading Avro to 1.10.1 for this schema::
```
{
"type": "record",
"name": "EventData",
"doc": "event data",
"fields": [
{"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
"string"}, "default": null }
]
}
```
```
org.apache.avro.AvroTypeException: Invalid default for field
ARRAY_WITH_DEFAULT: null not a {"type":"array","items":"string"}
at org.apache.avro.Schema.validateDefault(Schema.java:1571)
at org.apache.avro.Schema.access$500(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:544)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
```
Related Avro ticket: https://issues.apache.org/jira/browse/AVRO-2035
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554984)
Time Spent: 0.5h (was: 20m)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
>
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=554985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554985
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 23:55
Start Date: 19/Feb/21 23:55
Worklog Time Spent: 10m
Work Description: wangyum closed pull request #1994:
URL: https://github.com/apache/hive/pull/1994
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554985)
Time Spent: 40m (was: 0.5h)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-13825) Using JOIN in 2 tables that has same path locations, but different colum names fail wtih an error exception
[
https://issues.apache.org/jira/browse/HIVE-13825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-13825:
--
Assignee: (was: Vihang Karajgaonkar)
> Using JOIN in 2 tables that has same path locations, but different colum
> names fail wtih an error exception
> ---
>
> Key: HIVE-13825
> URL: https://issues.apache.org/jira/browse/HIVE-13825
> Project: Hive
> Issue Type: Improvement
>Reporter: Sergio Peña
>Priority: Major
>
> The following scenario of 2 tables with same locations cannot be used on a
> JOIN query:
> {noformat}
> hive> create table t1 (a string, b string) location
> '/user/hive/warehouse/test1';
> OK
> hive> create table t2 (c string, d string) location
> '/user/hive/warehouse/test1';
> OK
> hive> select t1.a from t1 join t2 on t1.a = t2.c;
> ...
> 2016-05-23 16:39:57 Starting to launch local task to process map join;
> maximum memory = 477102080
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-4
> Logs:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> {noformat}
> The logs contain this error exception:
> {noformat}
> 2016-05-23T16:39:58,163 ERROR [main]: mr.MapredLocalTask (:()) - Hive Runtime
> Error: Map local work failed
> java.lang.RuntimeException: cannot find field a from [0:c, 1:d]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.BaseStructObjectInspector.getStructFieldRef(BaseStructObjectInspector.java:133)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:973)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:999)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:75)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:499)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:403)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:383)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:751)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24522) Support DATE and VARCHAR column type in Kudu SerDe
[ https://issues.apache.org/jira/browse/HIVE-24522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned HIVE-24522: -- Assignee: Grant Henke > Support DATE and VARCHAR column type in Kudu SerDe > -- > > Key: HIVE-24522 > URL: https://issues.apache.org/jira/browse/HIVE-24522 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 2.0.0 >Reporter: Greg Solovyev >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > DATE type was added to Kudu in Kudu 1.12 (KUDU-2632) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23882) Compiler extensions for MJ probe optimization
[ https://issues.apache.org/jira/browse/HIVE-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-23882. - Fix Version/s: 4.0.0 Resolution: Fixed Pushed to master. Thanks, Panos & Zoltan! > Compiler extensions for MJ probe optimization > - > > Key: HIVE-23882 > URL: https://issues.apache.org/jira/browse/HIVE-23882 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > In probe we cannot currently support Key expressions (on the big table Side) > as ORC CVs Probe directly the smalltable HT (there is no expr evaluation at > that level). > TezCompiler should take this into account when picking MJs to push probe > details, In this ticket we extend probedecode compiler logic to: > * Validate that MJ is a single Key join, where the bigTable keyCol is only a > ExprNodeColumnDesc > * For the selected MJ use backtracking logic to find the orignal TS key it > maps to (on the same vertex) - this actually revealed several missed or wrong > optimizations (updated q outs) > * Finally, extend the optimization logic to check if there a type Cast > between src and destination (as the types have to match in the probe case) > and only use it on the LLAP mode for now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24522) Support DATE and VARCHAR column type in Kudu SerDe
[ https://issues.apache.org/jira/browse/HIVE-24522?focusedWorklogId=554932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554932 ] ASF GitHub Bot logged work on HIVE-24522: - Author: ASF GitHub Bot Created on: 19/Feb/21 21:09 Start Date: 19/Feb/21 21:09 Worklog Time Spent: 10m Work Description: granthenke opened a new pull request #1997: URL: https://github.com/apache/hive/pull/1997 This patch updates the Kudu version to the latest and adds support for the new DATE and VARCHAR column types. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554932) Remaining Estimate: 0h Time Spent: 10m > Support DATE and VARCHAR column type in Kudu SerDe > -- > > Key: HIVE-24522 > URL: https://issues.apache.org/jira/browse/HIVE-24522 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 2.0.0 >Reporter: Greg Solovyev >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > DATE type was added to Kudu in Kudu 1.12 (KUDU-2632) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24522) Support DATE and VARCHAR column type in Kudu SerDe
[ https://issues.apache.org/jira/browse/HIVE-24522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24522: -- Labels: pull-request-available (was: ) > Support DATE and VARCHAR column type in Kudu SerDe > -- > > Key: HIVE-24522 > URL: https://issues.apache.org/jira/browse/HIVE-24522 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 2.0.0 >Reporter: Greg Solovyev >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > DATE type was added to Kudu in Kudu 1.12 (KUDU-2632) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24775) Incorrect null handling when rebuilding Materialized view incrementally
[
https://issues.apache.org/jira/browse/HIVE-24775?focusedWorklogId=554924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554924
]
ASF GitHub Bot logged work on HIVE-24775:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 20:51
Start Date: 19/Feb/21 20:51
Worklog Time Spent: 10m
Work Description: jcamachor commented on a change in pull request #1995:
URL: https://github.com/apache/hive/pull/1995#discussion_r579468586
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveAggregateIncrementalRewritingRule.java
##
@@ -99,31 +100,61 @@ public void onMatch(RelOptRuleCall call) {
final RexBuilder rexBuilder =
agg.getCluster().getRexBuilder();
// 1) First branch is query, second branch is MV
-final RelNode joinLeftInput = union.getInput(1);
+RelNode joinLeftInput = union.getInput(1);
final RelNode joinRightInput = union.getInput(0);
-// 2) Build conditions for join and filter and start adding
+
+// 2) Introduce a Project on top of MV scan having all columns from the
view plus a boolean literal which indicates
+// whether the row with the key values coming from the joinRightInput
exists in the view:
+// - true means exist
+// - null means not exists
+// Project also needed to encapsulate the view scan by a subquery -> this
is required by
+// CalcitePlanner.fixUpASTAggregateIncrementalRebuild
+List mvCols = new
ArrayList<>(joinLeftInput.getRowType().getFieldCount());
+for (int i = 0; i < joinLeftInput.getRowType().getFieldCount(); ++i) {
+ mvCols.add(rexBuilder.makeInputRef(
+ joinLeftInput.getRowType().getFieldList().get(i).getType(), i));
+}
+mvCols.add(rexBuilder.makeLiteral(true));
+
+joinLeftInput = call.builder()
+.push(joinLeftInput)
+.project(mvCols)
+.build();
+
+// 3) Build conditions for join and start adding
// expressions for project operator
List projExprs = new ArrayList<>();
List joinConjs = new ArrayList<>();
-List filterConjs = new ArrayList<>();
int groupCount = agg.getGroupCount();
int totalCount = agg.getGroupCount() + agg.getAggCallList().size();
-for (int leftPos = 0, rightPos = totalCount;
+for (int leftPos = 0, rightPos = totalCount + 1;
leftPos < groupCount; leftPos++, rightPos++) {
RexNode leftRef = rexBuilder.makeInputRef(
joinLeftInput.getRowType().getFieldList().get(leftPos).getType(),
leftPos);
RexNode rightRef = rexBuilder.makeInputRef(
joinRightInput.getRowType().getFieldList().get(leftPos).getType(),
rightPos);
projExprs.add(rightRef);
- joinConjs.add(rexBuilder.makeCall(SqlStdOperatorTable.EQUALS,
- ImmutableList.of(leftRef, rightRef)));
- filterConjs.add(rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL,
- ImmutableList.of(leftRef)));
+
+ RexNode nsEqExpr =
rexBuilder.makeCall(SqlStdOperatorTable.IS_NOT_DISTINCT_FROM,
+ ImmutableList.of(leftRef, rightRef));
+ joinConjs.add(nsEqExpr);
}
-// 3) Add the expressions that correspond to the aggregation
+
+// 4) Create join node
+RexNode joinCond = RexUtil.composeConjunction(rexBuilder, joinConjs);
+RelNode join = call.builder()
+.push(joinLeftInput)
+.push(joinRightInput)
+.join(JoinRelType.RIGHT, joinCond)
+.build();
+
+int flagIndex = joinLeftInput.getRowType().getFieldCount() - 1;
+RexNode flagNode = rexBuilder.makeInputRef(
+join.getRowType().getFieldList().get(flagIndex).getType(),
flagIndex);
+
+// 5) Add the expressions that correspond to the aggregation
// functions
-RexNode caseFilterCond = RexUtil.composeConjunction(rexBuilder,
filterConjs);
-for (int i = 0, leftPos = groupCount, rightPos = totalCount + groupCount;
+for (int i = 0, leftPos = groupCount, rightPos = totalCount + 1 +
groupCount;
leftPos < totalCount; i++, leftPos++, rightPos++) {
// case when mv2.deptno IS NULL AND mv2.deptname IS NULL then s else
source.s + mv2.s end
Review comment:
This case statement should rely on the flag too?
```
// case mv2.flag IS NULL then s else source.s + mv2.s end
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554924)
Time Spent: 1h 10m (was: 1h)
> Incorrect null handling when rebuilding Materialized view incrementally
>
[jira] [Work logged] (HIVE-24743) [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2
[ https://issues.apache.org/jira/browse/HIVE-24743?focusedWorklogId=554916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554916 ] ASF GitHub Bot logged work on HIVE-24743: - Author: ASF GitHub Bot Created on: 19/Feb/21 20:36 Start Date: 19/Feb/21 20:36 Worklog Time Spent: 10m Work Description: kishendas commented on a change in pull request #1956: URL: https://github.com/apache/hive/pull/1956#discussion_r579461372 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java ## @@ -1497,8 +1497,8 @@ GetPartitionsPsWithAuthResponse listPartitionsWithAuthInfoRequest(GetPartitionsP * @throws MetaException error accessing the RDBMS. * @throws TException thrift transport error */ - List getPartitionsByNames(String db_name, String tbl_name, - List part_names) throws NoSuchObjectException, MetaException, TException; + List getPartitionsByNames(String db_name, String tbl_name, List part_names, Review comment: I am not changing HMS APIs here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554916) Time Spent: 2h 10m (was: 2h) > [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2 > -- > > Key: HIVE-24743 > URL: https://issues.apache.org/jira/browse/HIVE-24743 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > As part of ( HIVE-23821: Send tableId in request for all the new HMS > get_partition APIs ) we added logic to send tableId in the request for > several get_partition APIs, but looks like it was missed out for > getPartitionsByNames. TableId and validWriteIdList are used to maintain > consistency, when HMS API response is being served from a remote cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24743) [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2
[ https://issues.apache.org/jira/browse/HIVE-24743?focusedWorklogId=554915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554915 ] ASF GitHub Bot logged work on HIVE-24743: - Author: ASF GitHub Bot Created on: 19/Feb/21 20:34 Start Date: 19/Feb/21 20:34 Worklog Time Spent: 10m Work Description: yongzhi commented on a change in pull request #1956: URL: https://github.com/apache/hive/pull/1956#discussion_r579460370 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java ## @@ -1497,8 +1497,8 @@ GetPartitionsPsWithAuthResponse listPartitionsWithAuthInfoRequest(GetPartitionsP * @throws MetaException error accessing the RDBMS. * @throws TException thrift transport error */ - List getPartitionsByNames(String db_name, String tbl_name, - List part_names) throws NoSuchObjectException, MetaException, TException; + List getPartitionsByNames(String db_name, String tbl_name, List part_names, Review comment: How about old version clients? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554915) Time Spent: 2h (was: 1h 50m) > [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2 > -- > > Key: HIVE-24743 > URL: https://issues.apache.org/jira/browse/HIVE-24743 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > As part of ( HIVE-23821: Send tableId in request for all the new HMS > get_partition APIs ) we added logic to send tableId in the request for > several get_partition APIs, but looks like it was missed out for > getPartitionsByNames. TableId and validWriteIdList are used to maintain > consistency, when HMS API response is being served from a remote cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24522) Support DATE and VARCHAR column type in Kudu SerDe
[ https://issues.apache.org/jira/browse/HIVE-24522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated HIVE-24522: --- Summary: Support DATE and VARCHAR column type in Kudu SerDe (was: Support DATE column type in Kudu SerDe) > Support DATE and VARCHAR column type in Kudu SerDe > -- > > Key: HIVE-24522 > URL: https://issues.apache.org/jira/browse/HIVE-24522 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 2.0.0 >Reporter: Greg Solovyev >Priority: Major > > DATE type was added to Kudu in Kudu 1.12 (KUDU-2632) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24743) [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2
[ https://issues.apache.org/jira/browse/HIVE-24743?focusedWorklogId=554886=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554886 ] ASF GitHub Bot logged work on HIVE-24743: - Author: ASF GitHub Bot Created on: 19/Feb/21 19:35 Start Date: 19/Feb/21 19:35 Worklog Time Spent: 10m Work Description: kishendas commented on a change in pull request #1956: URL: https://github.com/apache/hive/pull/1956#discussion_r579428630 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java ## @@ -1497,8 +1497,8 @@ GetPartitionsPsWithAuthResponse listPartitionsWithAuthInfoRequest(GetPartitionsP * @throws MetaException error accessing the RDBMS. * @throws TException thrift transport error */ - List getPartitionsByNames(String db_name, String tbl_name, - List part_names) throws NoSuchObjectException, MetaException, TException; + List getPartitionsByNames(String db_name, String tbl_name, List part_names, Review comment: No, we want everyone to send validWriteIdList and tableId going forward. So, creating new interfaces doesn't work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554886) Time Spent: 1h 50m (was: 1h 40m) > [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2 > -- > > Key: HIVE-24743 > URL: https://issues.apache.org/jira/browse/HIVE-24743 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > As part of ( HIVE-23821: Send tableId in request for all the new HMS > get_partition APIs ) we added logic to send tableId in the request for > several get_partition APIs, but looks like it was missed out for > getPartitionsByNames. TableId and validWriteIdList are used to maintain > consistency, when HMS API response is being served from a remote cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24743) [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2
[ https://issues.apache.org/jira/browse/HIVE-24743?focusedWorklogId=554885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554885 ] ASF GitHub Bot logged work on HIVE-24743: - Author: ASF GitHub Bot Created on: 19/Feb/21 19:31 Start Date: 19/Feb/21 19:31 Worklog Time Spent: 10m Work Description: yongzhi commented on a change in pull request #1956: URL: https://github.com/apache/hive/pull/1956#discussion_r579426352 ## File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java ## @@ -1497,8 +1497,8 @@ GetPartitionsPsWithAuthResponse listPartitionsWithAuthInfoRequest(GetPartitionsP * @throws MetaException error accessing the RDBMS. * @throws TException thrift transport error */ - List getPartitionsByNames(String db_name, String tbl_name, - List part_names) throws NoSuchObjectException, MetaException, TException; + List getPartitionsByNames(String db_name, String tbl_name, List part_names, Review comment: Just change an interface may cause a backward compatible issue, maybe just add a new one and keeps old ones? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554885) Time Spent: 1h 40m (was: 1.5h) > [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2 > -- > > Key: HIVE-24743 > URL: https://issues.apache.org/jira/browse/HIVE-24743 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > As part of ( HIVE-23821: Send tableId in request for all the new HMS > get_partition APIs ) we added logic to send tableId in the request for > several get_partition APIs, but looks like it was missed out for > getPartitionsByNames. TableId and validWriteIdList are used to maintain > consistency, when HMS API response is being served from a remote cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24798) refactor TxnHandler.cleanupRecords to use predefined query strings
[ https://issues.apache.org/jira/browse/HIVE-24798?focusedWorklogId=554858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554858 ] ASF GitHub Bot logged work on HIVE-24798: - Author: ASF GitHub Bot Created on: 19/Feb/21 18:13 Start Date: 19/Feb/21 18:13 Worklog Time Spent: 10m Work Description: zchovan opened a new pull request #1996: URL: https://github.com/apache/hive/pull/1996 …y strings Change-Id: I596f286f586fceac7ba2b583659c334ad2ff1e52 ### What changes were proposed in this pull request? Some minor refactoring in TxnHandler.cleanupRecords. ### Why are the changes needed? In order avoid using StringBuilder to create query strings and use prepared statements for the queries. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No extra tests were added, run the existing tests to confirm no regression was introduced. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554858) Remaining Estimate: 0h Time Spent: 10m > refactor TxnHandler.cleanupRecords to use predefined query strings > -- > > Key: HIVE-24798 > URL: https://issues.apache.org/jira/browse/HIVE-24798 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > TxnHandler.cleanupRecords should use predefined query strings instead of > using a stringbuffer to build the delete queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24798) refactor TxnHandler.cleanupRecords to use predefined query strings
[ https://issues.apache.org/jira/browse/HIVE-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24798: -- Labels: pull-request-available (was: ) > refactor TxnHandler.cleanupRecords to use predefined query strings > -- > > Key: HIVE-24798 > URL: https://issues.apache.org/jira/browse/HIVE-24798 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > TxnHandler.cleanupRecords should use predefined query strings instead of > using a stringbuffer to build the delete queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24751) Kill trigger in workload manager fails with No privilege exception when authorization is disabled.
[
https://issues.apache.org/jira/browse/HIVE-24751?focusedWorklogId=554854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554854
]
ASF GitHub Bot logged work on HIVE-24751:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 18:04
Start Date: 19/Feb/21 18:04
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1964:
URL: https://github.com/apache/hive/pull/1964#discussion_r579374650
##
File path: service/src/java/org/apache/hive/service/server/KillQueryImpl.java
##
@@ -116,9 +117,21 @@ public static void killChildYarnJobs(Configuration conf,
String tag, String doAs
private static boolean isAdmin() {
boolean isAdmin = false;
-if (SessionState.get().getAuthorizerV2() != null) {
+SessionState ss = SessionState.get();
+if(!HiveConf.getBoolVar(ss.getConf(),
HiveConf.ConfVars.HIVE_AUTHORIZATION_ENABLED)) {
+ // If authorization is disabled, hs2 process owner should have kill
privileges
try {
-SessionState.get().getAuthorizerV2()
+String currentUser = ss.getUserName();
Review comment:
Can we try and add an unit test that hits this flow (if not already
exist)?
##
File path: service/src/java/org/apache/hive/service/server/KillQueryImpl.java
##
@@ -116,9 +117,21 @@ public static void killChildYarnJobs(Configuration conf,
String tag, String doAs
private static boolean isAdmin() {
boolean isAdmin = false;
-if (SessionState.get().getAuthorizerV2() != null) {
+SessionState ss = SessionState.get();
+if(!HiveConf.getBoolVar(ss.getConf(),
HiveConf.ConfVars.HIVE_AUTHORIZATION_ENABLED)) {
+ // If authorization is disabled, hs2 process owner should have kill
privileges
try {
-SessionState.get().getAuthorizerV2()
+String currentUser = ss.getUserName();
Review comment:
Can we try and add a unit test that hits this flow (if not already
exist)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554854)
Time Spent: 40m (was: 0.5h)
> Kill trigger in workload manager fails with No privilege exception when
> authorization is disabled.
> --
>
> Key: HIVE-24751
> URL: https://issues.apache.org/jira/browse/HIVE-24751
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> At present it is not checked whether authorization is enabled or not for Kill
> Query access.
> This causes Workload Manager thread to end up with No privilege Exception
> when trying to kill a query in an environment where authorization is disabled.
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:188)
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.lambda$scheduleWork$3(WorkloadManager.java:454)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.service.cli.HiveSQLException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:167)
> ... 6 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?focusedWorklogId=554851=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554851 ] ASF GitHub Bot logged work on HIVE-24645: - Author: ASF GitHub Bot Created on: 19/Feb/21 18:01 Start Date: 19/Feb/21 18:01 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #1876: URL: https://github.com/apache/hive/pull/1876 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554851) Time Spent: 1h 50m (was: 1h 40m) > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24751) Kill trigger in workload manager fails with No privilege exception when authorization is disabled.
[
https://issues.apache.org/jira/browse/HIVE-24751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-24751:
Summary: Kill trigger in workload manager fails with No privilege exception
when authorization is disabled. (was: Kill trigger in workload Manager fails
with NO privilege exception when authorization is disabled.)
> Kill trigger in workload manager fails with No privilege exception when
> authorization is disabled.
> --
>
> Key: HIVE-24751
> URL: https://issues.apache.org/jira/browse/HIVE-24751
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> At present it is not checked whether authorization is enabled or not for Kill
> Query access.
> This causes Workload Manager thread to end up with No privilege Exception
> when trying to kill a query in an environment where authorization is disabled.
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:188)
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.lambda$scheduleWork$3(WorkloadManager.java:454)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.service.cli.HiveSQLException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:167)
> ... 6 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24751) Kill trigger in workload Manager fails with NO privilege exception when authorization is disabled.
[
https://issues.apache.org/jira/browse/HIVE-24751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-24751:
Summary: Kill trigger in workload Manager fails with NO privilege exception
when authorization is disabled. (was: Workload Manager sees `No privilege`
exception even when authorization is not enabled)
> Kill trigger in workload Manager fails with NO privilege exception when
> authorization is disabled.
> --
>
> Key: HIVE-24751
> URL: https://issues.apache.org/jira/browse/HIVE-24751
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> At present it is not checked whether authorization is enabled or not for Kill
> Query access.
> This causes Workload Manager thread to end up with No privilege Exception
> when trying to kill a query in an environment where authorization is disabled.
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:188)
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.lambda$scheduleWork$3(WorkloadManager.java:454)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.service.cli.HiveSQLException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:167)
> ... 6 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24751) Workload Manager sees `No privilege` exception even when authorization is not enabled
[
https://issues.apache.org/jira/browse/HIVE-24751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan reassigned HIVE-24751:
---
Assignee: Nikhil Gupta (was: Sankar Hariappan)
> Workload Manager sees `No privilege` exception even when authorization is not
> enabled
> -
>
> Key: HIVE-24751
> URL: https://issues.apache.org/jira/browse/HIVE-24751
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Nikhil Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> At present it is not checked whether authorization is enabled or not for Kill
> Query access.
> This causes Workload Manager thread to end up with No privilege Exception
> when trying to kill a query in an environment where authorization is disabled.
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:188)
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.lambda$scheduleWork$3(WorkloadManager.java:454)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.service.cli.HiveSQLException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:167)
> ... 6 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24751) Workload Manager sees `No privilege` exception even when authorization is not enabled
[
https://issues.apache.org/jira/browse/HIVE-24751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan reassigned HIVE-24751:
---
Assignee: Sankar Hariappan (was: Nikhil Gupta)
> Workload Manager sees `No privilege` exception even when authorization is not
> enabled
> -
>
> Key: HIVE-24751
> URL: https://issues.apache.org/jira/browse/HIVE-24751
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Nikhil Gupta
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> At present it is not checked whether authorization is enabled or not for Kill
> Query access.
> This causes Workload Manager thread to end up with No privilege Exception
> when trying to kill a query in an environment where authorization is disabled.
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:188)
> at
> org.apache.hadoop.hive.ql.exec.tez.WorkloadManager.lambda$scheduleWork$3(WorkloadManager.java:454)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hive.service.cli.HiveSQLException: No privilege
> at
> org.apache.hive.service.server.KillQueryImpl.killQuery(KillQueryImpl.java:167)
> ... 6 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24259) [CachedStore] Optimise get all constraint api
[
https://issues.apache.org/jira/browse/HIVE-24259?focusedWorklogId=554847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554847
]
ASF GitHub Bot logged work on HIVE-24259:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 17:49
Start Date: 19/Feb/21 17:49
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r579358426
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern)
{
}
return false;
}
+
+ public static boolean
validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String
dbName,
Review comment:
It is not used anywhere other than below method. We can remove it and
move these validations there.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern)
{
}
return false;
}
+
+ public static boolean
validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String
dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents &&
rawStore.isActiveTransaction());
+ }
+
+ public static boolean
validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String
catName,
Review comment:
The method name is too long and not readable. Shall make it
shouldGetConstraintFromRawStore().
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -2554,8 +2554,7 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
-if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents &&
rawStore.isActiveTransaction()) || !sharedCache
-.isTableConstraintValid(catName, dbName, tblName)) {
+if
(validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(catName,dbName,tblName,canUseEvents,rawStore,sharedCache))
{
Review comment:
nit: Add space after each parameter.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern)
{
}
return false;
}
+
+ public static boolean
validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String
dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents &&
rawStore.isActiveTransaction());
+ }
+
+ public static boolean
validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String
catName,
Review comment:
Shall move this method as private method inside CachedStore to avoid
passing too many arguments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554847)
Time Spent: 5h 10m (was: 5h)
> [CachedStore] Optimise get all constraint api
> -
>
> Key: HIVE-24259
> URL: https://issues.apache.org/jira/browse/HIVE-24259
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashish Sharma
>Assignee: Ashish Sharma
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Description -
> Problem -
> 1. Redundant check if table is present or not
> 2. Currently in order to get all constraint form the cachedstore. 6 different
> call is made with in the cached store. Which led to 6 different call to raw
> store
> 3. If constraints are null and valid then also a redundant call is made to
> raw store.
>
> DOD
> 1. Create a flag which tell weather constraint snapshot is valid or not.
> 2. if incremental addition happen to any of the constraint mark snapshot as
> invalid and let updater thread to update cache with valid snapshot data.
> 3. Combine individual constraint call into one call for refresh and creation.
>
--
This message was sent by Atlassian Jira
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554843
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 17:39
Start Date: 19/Feb/21 17:39
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579358874
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/packages/HmsPackageRegistry.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql.packages;
+
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AddPackageRequest;
+import org.apache.hadoop.hive.metastore.api.DropPackageRequest;
+import org.apache.hadoop.hive.metastore.api.GetPackageRequest;
+import org.apache.hadoop.hive.metastore.api.Package;
+import org.apache.hive.hplsql.HplSqlSessionState;
+import org.apache.thrift.TException;
+
+public class HmsPackageRegistry implements PackageRegistry {
+ private final IMetaStoreClient msc;
+ private final HplSqlSessionState hplSqlSession;
+
+ public HmsPackageRegistry(IMetaStoreClient msc, HplSqlSessionState
hplSqlSession) {
+this.msc = msc;
+this.hplSqlSession = hplSqlSession;
+ }
+
+ @Override
+ public Optional getPackage(String name) {
+try {
+ Package pkg = msc.findPackage(request(name));
+ return pkg == null
+ ? Optional.empty()
+ : Optional.of(pkg.getHeader() + ";\n" + pkg.getBody());
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageHeader(String name, String header, boolean replace)
{
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && !replace)
+throw new RuntimeException("Package " + name + " already exists");
+ msc.addPackage(makePackage(name, header, ""));
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageBody(String name, String body, boolean replace) {
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && StringUtils.isNotEmpty(existing.getBody()) &&
!replace)
+throw new RuntimeException("Package body " + name + " already exists");
+ if (existing == null) {
+msc.addPackage(makePackage(name, "", body));
Review comment:
Ok, let's go with not allowing headerless packages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554843)
Time Spent: 6h 50m (was: 6h 40m)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554837
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 17:27
Start Date: 19/Feb/21 17:27
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579350072
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/packages/HmsPackageRegistry.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql.packages;
+
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AddPackageRequest;
+import org.apache.hadoop.hive.metastore.api.DropPackageRequest;
+import org.apache.hadoop.hive.metastore.api.GetPackageRequest;
+import org.apache.hadoop.hive.metastore.api.Package;
+import org.apache.hive.hplsql.HplSqlSessionState;
+import org.apache.thrift.TException;
+
+public class HmsPackageRegistry implements PackageRegistry {
+ private final IMetaStoreClient msc;
+ private final HplSqlSessionState hplSqlSession;
+
+ public HmsPackageRegistry(IMetaStoreClient msc, HplSqlSessionState
hplSqlSession) {
+this.msc = msc;
+this.hplSqlSession = hplSqlSession;
+ }
+
+ @Override
+ public Optional getPackage(String name) {
+try {
+ Package pkg = msc.findPackage(request(name));
+ return pkg == null
+ ? Optional.empty()
+ : Optional.of(pkg.getHeader() + ";\n" + pkg.getBody());
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageHeader(String name, String header, boolean replace)
{
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && !replace)
+throw new RuntimeException("Package " + name + " already exists");
+ msc.addPackage(makePackage(name, header, ""));
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageBody(String name, String body, boolean replace) {
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && StringUtils.isNotEmpty(existing.getBody()) &&
!replace)
+throw new RuntimeException("Package body " + name + " already exists");
+ if (existing == null) {
+msc.addPackage(makePackage(name, "", body));
Review comment:
But our design does not allow creating a body, then later creating a
header for it. `createPackageHeader` queries HMS with the package name. If it
finds the package, it always overrides the body.
I am fine with either
- not allowing headerless packages
- allowing to create a header later for a package
However, current design does neither of them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554837)
Time Spent: 6h 40m (was: 6.5h)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554834
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 17:25
Start Date: 19/Feb/21 17:25
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579350072
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/packages/HmsPackageRegistry.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql.packages;
+
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AddPackageRequest;
+import org.apache.hadoop.hive.metastore.api.DropPackageRequest;
+import org.apache.hadoop.hive.metastore.api.GetPackageRequest;
+import org.apache.hadoop.hive.metastore.api.Package;
+import org.apache.hive.hplsql.HplSqlSessionState;
+import org.apache.thrift.TException;
+
+public class HmsPackageRegistry implements PackageRegistry {
+ private final IMetaStoreClient msc;
+ private final HplSqlSessionState hplSqlSession;
+
+ public HmsPackageRegistry(IMetaStoreClient msc, HplSqlSessionState
hplSqlSession) {
+this.msc = msc;
+this.hplSqlSession = hplSqlSession;
+ }
+
+ @Override
+ public Optional getPackage(String name) {
+try {
+ Package pkg = msc.findPackage(request(name));
+ return pkg == null
+ ? Optional.empty()
+ : Optional.of(pkg.getHeader() + ";\n" + pkg.getBody());
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageHeader(String name, String header, boolean replace)
{
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && !replace)
+throw new RuntimeException("Package " + name + " already exists");
+ msc.addPackage(makePackage(name, header, ""));
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageBody(String name, String body, boolean replace) {
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && StringUtils.isNotEmpty(existing.getBody()) &&
!replace)
+throw new RuntimeException("Package body " + name + " already exists");
+ if (existing == null) {
+msc.addPackage(makePackage(name, "", body));
Review comment:
But our design does not allow creating a body, then later creating a
header for it. `createPackageHeader` queries HMS with the package name. If it
finds the package, it always overrides the body.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554834)
Time Spent: 6.5h (was: 6h 20m)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554833=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554833
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 17:25
Start Date: 19/Feb/21 17:25
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579350072
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/packages/HmsPackageRegistry.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql.packages;
+
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AddPackageRequest;
+import org.apache.hadoop.hive.metastore.api.DropPackageRequest;
+import org.apache.hadoop.hive.metastore.api.GetPackageRequest;
+import org.apache.hadoop.hive.metastore.api.Package;
+import org.apache.hive.hplsql.HplSqlSessionState;
+import org.apache.thrift.TException;
+
+public class HmsPackageRegistry implements PackageRegistry {
+ private final IMetaStoreClient msc;
+ private final HplSqlSessionState hplSqlSession;
+
+ public HmsPackageRegistry(IMetaStoreClient msc, HplSqlSessionState
hplSqlSession) {
+this.msc = msc;
+this.hplSqlSession = hplSqlSession;
+ }
+
+ @Override
+ public Optional getPackage(String name) {
+try {
+ Package pkg = msc.findPackage(request(name));
+ return pkg == null
+ ? Optional.empty()
+ : Optional.of(pkg.getHeader() + ";\n" + pkg.getBody());
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageHeader(String name, String header, boolean replace)
{
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && !replace)
+throw new RuntimeException("Package " + name + " already exists");
+ msc.addPackage(makePackage(name, header, ""));
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageBody(String name, String body, boolean replace) {
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && StringUtils.isNotEmpty(existing.getBody()) &&
!replace)
+throw new RuntimeException("Package body " + name + " already exists");
+ if (existing == null) {
+msc.addPackage(makePackage(name, "", body));
Review comment:
But our design does not allow creating a body than creating a header for
it. `createPackageHeader` queries HMS with the package name. If it finds the
package, it always overrides the body.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554833)
Time Spent: 6h 20m (was: 6h 10m)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24726) Track required data for cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24726?focusedWorklogId=554793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554793 ] ASF GitHub Bot logged work on HIVE-24726: - Author: ASF GitHub Bot Created on: 19/Feb/21 15:51 Start Date: 19/Feb/21 15:51 Worklog Time Spent: 10m Work Description: szlta merged pull request #1961: URL: https://github.com/apache/hive/pull/1961 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554793) Time Spent: 50m (was: 40m) > Track required data for cache hydration > --- > > Key: HIVE-24726 > URL: https://issues.apache.org/jira/browse/HIVE-24726 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24726) Track required data for cache hydration
[ https://issues.apache.org/jira/browse/HIVE-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita resolved HIVE-24726. --- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master, thanks [~asinkovits] > Track required data for cache hydration > --- > > Key: HIVE-24726 > URL: https://issues.apache.org/jira/browse/HIVE-24726 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24725) Collect top priority items from llap cache policy
[ https://issues.apache.org/jira/browse/HIVE-24725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita resolved HIVE-24725. --- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master, thanks [~asinkovits] > Collect top priority items from llap cache policy > - > > Key: HIVE-24725 > URL: https://issues.apache.org/jira/browse/HIVE-24725 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24725) Collect top priority items from llap cache policy
[ https://issues.apache.org/jira/browse/HIVE-24725?focusedWorklogId=554791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554791 ] ASF GitHub Bot logged work on HIVE-24725: - Author: ASF GitHub Bot Created on: 19/Feb/21 15:28 Start Date: 19/Feb/21 15:28 Worklog Time Spent: 10m Work Description: szlta merged pull request #1947: URL: https://github.com/apache/hive/pull/1947 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554791) Time Spent: 50m (was: 40m) > Collect top priority items from llap cache policy > - > > Key: HIVE-24725 > URL: https://issues.apache.org/jira/browse/HIVE-24725 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24693) Convert timestamps to zoned times without string operations
[
https://issues.apache.org/jira/browse/HIVE-24693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-24693.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Pushed to master. Thanks [~pgaref] for the review!
> Convert timestamps to zoned times without string operations
> ---
>
> Key: HIVE-24693
> URL: https://issues.apache.org/jira/browse/HIVE-24693
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Parquet {{DataWriteableWriter}} relias on {{NanoTimeUtils}} to convert a
> timestamp object into a binary value. The way in which it does this,... it
> calls {{toString()}} on the timestamp object, and then parses the String.
> This particular timestamp do not carry a timezone, so the string is something
> like:
> {{2021-21-03 12:32:23....}}
> The parse code tries to parse the string assuming there is a time zone, and
> if not, falls-back and applies the provided "default time zone". As was
> noted in [HIVE-24353], if something fails to parse, it is very expensive to
> try to parse again. So, for each timestamp in the Parquet file, it:
> * Builds a string from the time stamp
> * Parses it (throws an exception, parses again)
> There is no need to do this kind of string manipulations/parsing, it should
> just be using the epoch millis/seconds/time stored internal to the Timestamp
> object.
> {code:java}
> // Converts Timestamp to TimestampTZ.
> public static TimestampTZ convert(Timestamp ts, ZoneId defaultTimeZone) {
> return parse(ts.toString(), defaultTimeZone);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24693) Convert timestamps to zoned times without string operations
[
https://issues.apache.org/jira/browse/HIVE-24693?focusedWorklogId=554785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554785
]
ASF GitHub Bot logged work on HIVE-24693:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 14:49
Start Date: 19/Feb/21 14:49
Worklog Time Spent: 10m
Work Description: belugabehr merged pull request #1938:
URL: https://github.com/apache/hive/pull/1938
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554785)
Time Spent: 6h 10m (was: 6h)
> Convert timestamps to zoned times without string operations
> ---
>
> Key: HIVE-24693
> URL: https://issues.apache.org/jira/browse/HIVE-24693
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Parquet {{DataWriteableWriter}} relias on {{NanoTimeUtils}} to convert a
> timestamp object into a binary value. The way in which it does this,... it
> calls {{toString()}} on the timestamp object, and then parses the String.
> This particular timestamp do not carry a timezone, so the string is something
> like:
> {{2021-21-03 12:32:23....}}
> The parse code tries to parse the string assuming there is a time zone, and
> if not, falls-back and applies the provided "default time zone". As was
> noted in [HIVE-24353], if something fails to parse, it is very expensive to
> try to parse again. So, for each timestamp in the Parquet file, it:
> * Builds a string from the time stamp
> * Parses it (throws an exception, parses again)
> There is no need to do this kind of string manipulations/parsing, it should
> just be using the epoch millis/seconds/time stored internal to the Timestamp
> object.
> {code:java}
> // Converts Timestamp to TimestampTZ.
> public static TimestampTZ convert(Timestamp ts, ZoneId defaultTimeZone) {
> return parse(ts.toString(), defaultTimeZone);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24775) Incorrect null handling when rebuilding Materialized view incrementally
[
https://issues.apache.org/jira/browse/HIVE-24775?focusedWorklogId=554765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554765
]
ASF GitHub Bot logged work on HIVE-24775:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 13:59
Start Date: 19/Feb/21 13:59
Worklog Time Spent: 10m
Work Description: kasakrisz commented on pull request #1981:
URL: https://github.com/apache/hive/pull/1981#issuecomment-782092085
Addressed review comments in #1995 since a completely new approach was
implemented.
Closing this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554765)
Time Spent: 50m (was: 40m)
> Incorrect null handling when rebuilding Materialized view incrementally
> ---
>
> Key: HIVE-24775
> URL: https://issues.apache.org/jira/browse/HIVE-24775
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE t1 (a int, b varchar(256), c decimal(10,2), d int) STORED AS orc
> TBLPROPERTIES ('transactional'='true');
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.77, 7),
> (1, 'calvin', 978.76, 3),
> (1, 'charlie', 9.8, 1);
> CREATE MATERIALIZED VIEW mat1 TBLPROPERTIES ('transactional'='true') AS
> SELECT a, b, sum(d)
> FROM t1
> WHERE c > 10.0
> GROUP BY a, b;
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.88, 8),
> (1, 'charlie', 15.8, 1);
> ALTER MATERIALIZED VIEW mat1 REBUILD;
> SELECT * FROM mat1
> ORDER BY a, b;
> {code}
> View contains:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 8
> NULL null_value 7
> {code}
> but it should contain:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 15
> {code}
> Rows with aggregate key columns having NULL values are not aggregated because
> incremental materialized view rebuild plan is altered by
> [applyPreJoinOrderingTransforms|https://github.com/apache/hive/blob/76732ad27e139fbdef25b820a07cf35934771083/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1975]:
> IS NOT NULL filter added for each of these columns on top of the view scan
> when joining with the branch pulls the rows inserted after the last rebuild:
> {code}
> HiveProject($f0=[$3], $f1=[$4], $f2=[CASE(AND(IS NULL($0), IS NULL($1)), $5,
> +($5, $2))])
> HiveFilter(condition=[OR(AND(IS NULL($0), IS NULL($1)), AND(=($0, $3),
> =($1, $4)))])
> HiveJoin(condition=[AND(=($0, $3), =($1, $4))], joinType=[right],
> algorithm=[none], cost=[not available])
> HiveProject(a=[$0], b=[$1], _c2=[$2])
> HiveFilter(condition=[AND(IS NOT NULL($0), IS NOT NULL($1))])
> HiveTableScan(table=[[default, mat1]], table:alias=[default.mat1])
> HiveProject(a=[$0], b=[$1], $f2=[$2])
> HiveAggregate(group=[{0, 1}], agg#0=[sum($3)])
> HiveFilter(condition=[AND(<(1, $6.writeid), >($2, 10))])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24775) Incorrect null handling when rebuilding Materialized view incrementally
[
https://issues.apache.org/jira/browse/HIVE-24775?focusedWorklogId=554766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554766
]
ASF GitHub Bot logged work on HIVE-24775:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 13:59
Start Date: 19/Feb/21 13:59
Worklog Time Spent: 10m
Work Description: kasakrisz closed pull request #1981:
URL: https://github.com/apache/hive/pull/1981
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554766)
Time Spent: 1h (was: 50m)
> Incorrect null handling when rebuilding Materialized view incrementally
> ---
>
> Key: HIVE-24775
> URL: https://issues.apache.org/jira/browse/HIVE-24775
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE t1 (a int, b varchar(256), c decimal(10,2), d int) STORED AS orc
> TBLPROPERTIES ('transactional'='true');
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.77, 7),
> (1, 'calvin', 978.76, 3),
> (1, 'charlie', 9.8, 1);
> CREATE MATERIALIZED VIEW mat1 TBLPROPERTIES ('transactional'='true') AS
> SELECT a, b, sum(d)
> FROM t1
> WHERE c > 10.0
> GROUP BY a, b;
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.88, 8),
> (1, 'charlie', 15.8, 1);
> ALTER MATERIALIZED VIEW mat1 REBUILD;
> SELECT * FROM mat1
> ORDER BY a, b;
> {code}
> View contains:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 8
> NULL null_value 7
> {code}
> but it should contain:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 15
> {code}
> Rows with aggregate key columns having NULL values are not aggregated because
> incremental materialized view rebuild plan is altered by
> [applyPreJoinOrderingTransforms|https://github.com/apache/hive/blob/76732ad27e139fbdef25b820a07cf35934771083/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1975]:
> IS NOT NULL filter added for each of these columns on top of the view scan
> when joining with the branch pulls the rows inserted after the last rebuild:
> {code}
> HiveProject($f0=[$3], $f1=[$4], $f2=[CASE(AND(IS NULL($0), IS NULL($1)), $5,
> +($5, $2))])
> HiveFilter(condition=[OR(AND(IS NULL($0), IS NULL($1)), AND(=($0, $3),
> =($1, $4)))])
> HiveJoin(condition=[AND(=($0, $3), =($1, $4))], joinType=[right],
> algorithm=[none], cost=[not available])
> HiveProject(a=[$0], b=[$1], _c2=[$2])
> HiveFilter(condition=[AND(IS NOT NULL($0), IS NOT NULL($1))])
> HiveTableScan(table=[[default, mat1]], table:alias=[default.mat1])
> HiveProject(a=[$0], b=[$1], $f2=[$2])
> HiveAggregate(group=[{0, 1}], agg#0=[sum($3)])
> HiveFilter(condition=[AND(<(1, $6.writeid), >($2, 10))])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24704) Ensure that all Operator column expressions refer to a column in the RowSchema
[
https://issues.apache.org/jira/browse/HIVE-24704?focusedWorklogId=554764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554764
]
ASF GitHub Bot logged work on HIVE-24704:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 13:57
Start Date: 19/Feb/21 13:57
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1929:
URL: https://github.com/apache/hive/pull/1929#discussion_r579203220
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/hooks/OperatorHealthCheckerHook.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.hooks;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.Stack;
+
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Sets;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.Operator;
+import org.apache.hadoop.hive.ql.exec.ReduceSinkOperator;
+import org.apache.hadoop.hive.ql.exec.RowSchema;
+import org.apache.hadoop.hive.ql.exec.ScriptOperator;
+import org.apache.hadoop.hive.ql.exec.SelectOperator;
+import org.apache.hadoop.hive.ql.exec.TableScanOperator;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker;
+import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher;
+import org.apache.hadoop.hive.ql.lib.SemanticDispatcher;
+import org.apache.hadoop.hive.ql.lib.SemanticGraphWalker;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.SemanticNodeProcessor;
+import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.plan.BaseWork;
+import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.GroupByDesc;
+import org.apache.hadoop.hive.ql.plan.MapWork;
+import org.apache.hadoop.hive.ql.plan.MapredWork;
+import org.apache.hadoop.hive.ql.plan.OperatorDesc;
+import org.apache.hadoop.hive.ql.plan.ReduceWork;
+import org.apache.hadoop.hive.ql.plan.SelectDesc;
+import org.apache.hadoop.hive.ql.plan.TezWork;
+
+/**
+ * Checks some operator quality rules.
+ *
+ * Checks whenever operator ids are not reused.
+ * Checks some level of expression/schema consistency
+ * Some sanity checks on SelectOperators
+ */
+public class OperatorHealthCheckerHook implements ExecuteWithHookContext {
+
+ static class UniqueOpIdChecker implements SemanticNodeProcessor {
+
+Map> opMap = new HashMap<>();
+
+@Override
+public Object process(Node nd, Stack stack, NodeProcessorCtx
procCtx, Object... nodeOutputs)
+throws SemanticException {
+
+ Operator op = (Operator) nd;
+ checkOperator(op);
+ String opKey = op.getOperatorId();
+ Operator found = opMap.get(opKey);
+ if (found != null) {
+throw new RuntimeException("operator id reuse found: " + opKey);
+ }
+ opMap.put(opKey, op);
+ return null;
+}
+ }
+
+ public static void checkOperator(Operator op) {
+OperatorDesc conf = op.getConf();
+Map exprMap = conf.getColumnExprMap();
+RowSchema schema = op.getSchema();
+
+chceckSchema(schema);
+if (op instanceof SelectOperator) {
+ checkSelectOperator((SelectOperator) op);
+}
+if (schema != null && exprMap != null) {
+ for (Entry c : exprMap.entrySet()) {
+if (c.getValue() instanceof ExprNodeConstantDesc) {
+ continue;
+}
+ColumnInfo ci = schema.getColumnInfo(c.getKey());
+if (c.getKey().startsWith("KEY.reducesinkkey")) {
Review comment:
opened HIVE-24799
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact
[jira] [Work logged] (HIVE-24775) Incorrect null handling when rebuilding Materialized view incrementally
[
https://issues.apache.org/jira/browse/HIVE-24775?focusedWorklogId=554763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554763
]
ASF GitHub Bot logged work on HIVE-24775:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 13:57
Start Date: 19/Feb/21 13:57
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #1995:
URL: https://github.com/apache/hive/pull/1995
### What changes were proposed in this pull request?
When transforming plan of materialized view rebuild to incremental rebuild
in case of the view definition has aggregate:
* Use null safe equality operators in join condition instead of equality
operators
* Introduce a Project on top of MV scan having all columns from the view
plus a boolean literal which indicates whether the row with the key values
coming from the joinRightInput exists in the view. Use this flag in
`CalcitePlanner.fixUpASTAggregateIncrementalRebuild` to decide which branch a
row should go: update or insert.
* Add null checks of aggregated values when calculated new aggregated values.
### Why are the changes needed?
Rows with null aggregate keys and aggregated values was not handled by
incremental MV rebuild and it could lead to data corruption.
### Does this PR introduce _any_ user-facing change?
Yes. Query result changes. See jira for example.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver
-Dqfile=materialized_view_create_rewrite_nulls.q,materialized_view_create_rewrite_4.q,materialized_view_create_rewrite_one_key_gby.q
-pl itests/qtest -Pitests
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554763)
Time Spent: 40m (was: 0.5h)
> Incorrect null handling when rebuilding Materialized view incrementally
> ---
>
> Key: HIVE-24775
> URL: https://issues.apache.org/jira/browse/HIVE-24775
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code}
> CREATE TABLE t1 (a int, b varchar(256), c decimal(10,2), d int) STORED AS orc
> TBLPROPERTIES ('transactional'='true');
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.77, 7),
> (1, 'calvin', 978.76, 3),
> (1, 'charlie', 9.8, 1);
> CREATE MATERIALIZED VIEW mat1 TBLPROPERTIES ('transactional'='true') AS
> SELECT a, b, sum(d)
> FROM t1
> WHERE c > 10.0
> GROUP BY a, b;
> INSERT INTO t1 VALUES
> (NULL, 'null_value', 100.88, 8),
> (1, 'charlie', 15.8, 1);
> ALTER MATERIALIZED VIEW mat1 REBUILD;
> SELECT * FROM mat1
> ORDER BY a, b;
> {code}
> View contains:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 8
> NULL null_value 7
> {code}
> but it should contain:
> {code}
> 1 calvin 3
> 1 charlie 1
> NULL null_value 15
> {code}
> Rows with aggregate key columns having NULL values are not aggregated because
> incremental materialized view rebuild plan is altered by
> [applyPreJoinOrderingTransforms|https://github.com/apache/hive/blob/76732ad27e139fbdef25b820a07cf35934771083/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L1975]:
> IS NOT NULL filter added for each of these columns on top of the view scan
> when joining with the branch pulls the rows inserted after the last rebuild:
> {code}
> HiveProject($f0=[$3], $f1=[$4], $f2=[CASE(AND(IS NULL($0), IS NULL($1)), $5,
> +($5, $2))])
> HiveFilter(condition=[OR(AND(IS NULL($0), IS NULL($1)), AND(=($0, $3),
> =($1, $4)))])
> HiveJoin(condition=[AND(=($0, $3), =($1, $4))], joinType=[right],
> algorithm=[none], cost=[not available])
> HiveProject(a=[$0], b=[$1], _c2=[$2])
> HiveFilter(condition=[AND(IS NOT NULL($0), IS NOT NULL($1))])
> HiveTableScan(table=[[default, mat1]], table:alias=[default.mat1])
> HiveProject(a=[$0], b=[$1], $f2=[$2])
> HiveAggregate(group=[{0, 1}], agg#0=[sum($3)])
> HiveFilter(condition=[AND(<(1, $6.writeid), >($2, 10))])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24799) Rewise RS rowschema/colExprMap consistency in case of mapjoins
[ https://issues.apache.org/jira/browse/HIVE-24799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-24799: --- > Rewise RS rowschema/colExprMap consistency in case of mapjoins > -- > > Key: HIVE-24799 > URL: https://issues.apache.org/jira/browse/HIVE-24799 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > I've seen some odd things around MJ - so I added an if to skip those errors > while I was fixing others. > IIRC this this issue seemed more serious; the code did "know" that the RS had > a bad schema and it went to the parent - I guess there was a reason to do > that - and I guess fixing it won't be easy either. > https://github.com/apache/hive/pull/1929#discussion_r579200496 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24704) Ensure that all Operator column expressions refer to a column in the RowSchema
[
https://issues.apache.org/jira/browse/HIVE-24704?focusedWorklogId=554760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554760
]
ASF GitHub Bot logged work on HIVE-24704:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 13:53
Start Date: 19/Feb/21 13:53
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1929:
URL: https://github.com/apache/hive/pull/1929#discussion_r579200496
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/hooks/OperatorHealthCheckerHook.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.hooks;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.Stack;
+
+import com.google.common.collect.Lists;
+
+import com.google.common.collect.Sets;
+import org.apache.hadoop.hive.ql.exec.ColumnInfo;
+import org.apache.hadoop.hive.ql.exec.Operator;
+import org.apache.hadoop.hive.ql.exec.ReduceSinkOperator;
+import org.apache.hadoop.hive.ql.exec.RowSchema;
+import org.apache.hadoop.hive.ql.exec.ScriptOperator;
+import org.apache.hadoop.hive.ql.exec.SelectOperator;
+import org.apache.hadoop.hive.ql.exec.TableScanOperator;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker;
+import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher;
+import org.apache.hadoop.hive.ql.lib.SemanticDispatcher;
+import org.apache.hadoop.hive.ql.lib.SemanticGraphWalker;
+import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.ql.lib.SemanticNodeProcessor;
+import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.plan.BaseWork;
+import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc;
+import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
+import org.apache.hadoop.hive.ql.plan.GroupByDesc;
+import org.apache.hadoop.hive.ql.plan.MapWork;
+import org.apache.hadoop.hive.ql.plan.MapredWork;
+import org.apache.hadoop.hive.ql.plan.OperatorDesc;
+import org.apache.hadoop.hive.ql.plan.ReduceWork;
+import org.apache.hadoop.hive.ql.plan.SelectDesc;
+import org.apache.hadoop.hive.ql.plan.TezWork;
+
+/**
+ * Checks some operator quality rules.
+ *
+ * Checks whenever operator ids are not reused.
+ * Checks some level of expression/schema consistency
+ * Some sanity checks on SelectOperators
+ */
+public class OperatorHealthCheckerHook implements ExecuteWithHookContext {
+
+ static class UniqueOpIdChecker implements SemanticNodeProcessor {
+
+Map> opMap = new HashMap<>();
+
+@Override
+public Object process(Node nd, Stack stack, NodeProcessorCtx
procCtx, Object... nodeOutputs)
+throws SemanticException {
+
+ Operator op = (Operator) nd;
+ checkOperator(op);
+ String opKey = op.getOperatorId();
+ Operator found = opMap.get(opKey);
+ if (found != null) {
+throw new RuntimeException("operator id reuse found: " + opKey);
+ }
+ opMap.put(opKey, op);
+ return null;
+}
+ }
+
+ public static void checkOperator(Operator op) {
+OperatorDesc conf = op.getConf();
+Map exprMap = conf.getColumnExprMap();
+RowSchema schema = op.getSchema();
+
+chceckSchema(schema);
+if (op instanceof SelectOperator) {
+ checkSelectOperator((SelectOperator) op);
+}
+if (schema != null && exprMap != null) {
+ for (Entry c : exprMap.entrySet()) {
+if (c.getValue() instanceof ExprNodeConstantDesc) {
+ continue;
+}
+ColumnInfo ci = schema.getColumnInfo(c.getKey());
+if (c.getKey().startsWith("KEY.reducesinkkey")) {
Review comment:
actually - this if is still here because in case of MJ the RS rowschema
is so bad that I kinda give up fixing it ; it needs to be taken care of
separately...
This is an automated message from the Apache Git Service.
To respond to the
[jira] [Assigned] (HIVE-24798) refactor TxnHandler.cleanupRecords to use predefined query strings
[ https://issues.apache.org/jira/browse/HIVE-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan reassigned HIVE-24798: > refactor TxnHandler.cleanupRecords to use predefined query strings > -- > > Key: HIVE-24798 > URL: https://issues.apache.org/jira/browse/HIVE-24798 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Minor > > TxnHandler.cleanupRecords should use predefined query strings instead of > using a stringbuffer to build the delete queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23882) Compiler extensions for MJ probe optimization
[ https://issues.apache.org/jira/browse/HIVE-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23882: -- Summary: Compiler extensions for MJ probe optimization (was: Compiler should skip MJ keyExpr for probe optimization) > Compiler extensions for MJ probe optimization > - > > Key: HIVE-23882 > URL: https://issues.apache.org/jira/browse/HIVE-23882 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > In probe we cannot currently support Key expressions (on the big table Side) > as ORC CVs Probe directly the smalltable HT (there is no expr evaluation at > that level). > TezCompiler should take this into account when picking MJs to push probe > details, In this ticket we extend probedecode compiler logic to: > * Validate that MJ is a single Key join, where the bigTable keyCol is only a > ExprNodeColumnDesc > * For the selected MJ use backtracking logic to find the orignal TS key it > maps to (on the same vertex) - this actually revealed several missed or wrong > optimizations (updated q outs) > * Finally, extend the optimization logic to check if there a type Cast > between src and destination (as the types have to match in the probe case) > and only use it on the LLAP mode for now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=554693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554693
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 10:16
Start Date: 19/Feb/21 10:16
Worklog Time Spent: 10m
Work Description: wangyum commented on pull request #1994:
URL: https://github.com/apache/hive/pull/1994#issuecomment-781979028
cc @sunchao
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554693)
Time Spent: 20m (was: 10m)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-24797:
--
Labels: pull-request-available (was: )
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?focusedWorklogId=554692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554692
]
ASF GitHub Bot logged work on HIVE-24797:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 10:16
Start Date: 19/Feb/21 10:16
Worklog Time Spent: 10m
Work Description: wangyum opened a new pull request #1994:
URL: https://github.com/apache/hive/pull/1994
### What changes were proposed in this pull request?
This pr disable validate default values when parsing Avro schemas.
### Why are the changes needed?
It will throw exception when upgrading Avro to 1.10.1 for this schema::
```
{
"type": "record",
"name": "EventData",
"doc": "event data",
"fields": [
{"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
"string"}, "default": null }
]
}
```
```
org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
null not a {"type":"array","items":"string"}
at org.apache.avro.Schema.validateDefault(Schema.java:1571)
at org.apache.avro.Schema.access$500(Schema.java:87)
at org.apache.avro.Schema$Field.(Schema.java:544)
at org.apache.avro.Schema.parse(Schema.java:1678)
at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
at
org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
```
Related Avro ticket: https://issues.apache.org/jira/browse/AVRO-2035
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554692)
Remaining Estimate: 0h
Time Spent: 10m
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
>
[jira] [Updated] (HIVE-24797) Disable validate default values when parsing Avro schemas
[
https://issues.apache.org/jira/browse/HIVE-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated HIVE-24797:
---
Summary: Disable validate default values when parsing Avro schemas (was:
Disable validate default values when parsing Avro schemas.)
> Disable validate default values when parsing Avro schemas
> -
>
> Key: HIVE-24797
> URL: https://issues.apache.org/jira/browse/HIVE-24797
> Project: Hive
> Issue Type: Bug
>Reporter: Yuming Wang
>Priority: Major
>
> It will throw exceptions when upgrading Avro to 1.10.1 for this schema:
> {code:json}
> {
> "type": "record",
> "name": "EventData",
> "doc": "event data",
> "fields": [
> {"name": "ARRAY_WITH_DEFAULT", "type": {"type": "array", "items":
> "string"}, "default": null }
> ]
> }
> {code}
> {noformat}
> org.apache.avro.AvroTypeException: Invalid default for field USERACTIONS:
> null not a {"type":"array","items":"string"}
> at org.apache.avro.Schema.validateDefault(Schema.java:1571)
> at org.apache.avro.Schema.access$500(Schema.java:87)
> at org.apache.avro.Schema$Field.(Schema.java:544)
> at org.apache.avro.Schema.parse(Schema.java:1678)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1425)
> at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFor(AvroSerdeUtils.java:287)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:170)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:187)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:107)
> at
> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
> at
> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:493)
> at
> org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:225)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554688
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 10:07
Start Date: 19/Feb/21 10:07
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579069892
##
File path: hplsql/src/main/java/org/apache/hive/hplsql/Exec.java
##
@@ -328,16 +334,16 @@ public String callStackPop() {
*/
public Var findVariable(String name) {
Var var;
-String name1 = name;
+String name1 = name.toUpperCase();
String name1a = null;
String name2 = null;
Scope cur = exec.currentScope;
Package pack;
Package packCallContext = exec.getPackageCallContext();
ArrayList qualified = exec.meta.splitIdentifier(name);
if (qualified != null) {
- name1 = qualified.get(0);
- name2 = qualified.get(1);
+ name1 = qualified.get(0).toUpperCase();
Review comment:
splitIdentifier() is also used with table names and column names, I
think it's better to leave column and table names alone as it would modify the
HiveQL queries and it's rather Hive's territory than HPLSQL.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554688)
Time Spent: 6h 10m (was: 6h)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24346) Store HPL/SQL packages into HMS
[
https://issues.apache.org/jira/browse/HIVE-24346?focusedWorklogId=554687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554687
]
ASF GitHub Bot logged work on HIVE-24346:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 10:04
Start Date: 19/Feb/21 10:04
Worklog Time Spent: 10m
Work Description: zeroflag commented on a change in pull request #1733:
URL: https://github.com/apache/hive/pull/1733#discussion_r579067983
##
File path:
hplsql/src/main/java/org/apache/hive/hplsql/packages/HmsPackageRegistry.java
##
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql.packages;
+
+import java.util.Optional;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.AddPackageRequest;
+import org.apache.hadoop.hive.metastore.api.DropPackageRequest;
+import org.apache.hadoop.hive.metastore.api.GetPackageRequest;
+import org.apache.hadoop.hive.metastore.api.Package;
+import org.apache.hive.hplsql.HplSqlSessionState;
+import org.apache.thrift.TException;
+
+public class HmsPackageRegistry implements PackageRegistry {
+ private final IMetaStoreClient msc;
+ private final HplSqlSessionState hplSqlSession;
+
+ public HmsPackageRegistry(IMetaStoreClient msc, HplSqlSessionState
hplSqlSession) {
+this.msc = msc;
+this.hplSqlSession = hplSqlSession;
+ }
+
+ @Override
+ public Optional getPackage(String name) {
+try {
+ Package pkg = msc.findPackage(request(name));
+ return pkg == null
+ ? Optional.empty()
+ : Optional.of(pkg.getHeader() + ";\n" + pkg.getBody());
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageHeader(String name, String header, boolean replace)
{
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && !replace)
+throw new RuntimeException("Package " + name + " already exists");
+ msc.addPackage(makePackage(name, header, ""));
+} catch (TException e) {
+ throw new RuntimeException(e.getCause());
+}
+ }
+
+ @Override
+ public void createPackageBody(String name, String body, boolean replace) {
+try {
+ Package existing = msc.findPackage(request(name));
+ if (existing != null && StringUtils.isNotEmpty(existing.getBody()) &&
!replace)
+throw new RuntimeException("Package body " + name + " already exists");
+ if (existing == null) {
+msc.addPackage(makePackage(name, "", body));
Review comment:
HPLSQL used to allow this in previous versions. Oracle also allows you
to create the body without the spec but the package is not usable until you
create the spec for it. Throwing an error at this point wouldn't be the same
either. We might want to put a check when getting the package, but it would
break the previous behaviour. I suspect the limitation in oracle is coming from
the fact that it compiles these two together but since this language is fully
interpreted we don't face this problem. An other option would be to print out a
warning message. Or just leave it as it is since this is how it used to work.
Which one do you think is better?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554687)
Time Spent: 6h (was: 5h 50m)
> Store HPL/SQL packages into HMS
> ---
>
> Key: HIVE-24346
> URL: https://issues.apache.org/jira/browse/HIVE-24346
> Project: Hive
> Issue Type: Sub-task
> Components: hpl/sql, Metastore
>Reporter: Attila Magyar
>Assignee: Attila
[jira] [Work logged] (HIVE-24792) Potential thread leak in Operation
[ https://issues.apache.org/jira/browse/HIVE-24792?focusedWorklogId=554665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554665 ] ASF GitHub Bot logged work on HIVE-24792: - Author: ASF GitHub Bot Created on: 19/Feb/21 09:07 Start Date: 19/Feb/21 09:07 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on pull request #1992: URL: https://github.com/apache/hive/pull/1992#issuecomment-781939279 Move the operation log cleanup logic into a standalone method... @prasanthj could you please take a look? thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554665) Time Spent: 40m (was: 0.5h) > Potential thread leak in Operation > -- > > Key: HIVE-24792 > URL: https://issues.apache.org/jira/browse/HIVE-24792 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The _scheduledExecutorService_ in _Operation_ does not shut down after > scheduling delay operationlog cleanup, which may result to thread leak in > hiveserver2... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23834) [CachedStore] Add flag in TableWrapper in CacheStore to check if constraints are set or not
[ https://issues.apache.org/jira/browse/HIVE-23834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma resolved HIVE-23834. -- Resolution: Fixed > [CachedStore] Add flag in TableWrapper in CacheStore to check if constraints > are set or not > --- > > Key: HIVE-23834 > URL: https://issues.apache.org/jira/browse/HIVE-23834 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Adesh Kumar Rao >Assignee: Adesh Kumar Rao >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24785) Fix HIVE_COMPACTOR_COMPACT_MM property
[ https://issues.apache.org/jira/browse/HIVE-24785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga resolved HIVE-24785. Fix Version/s: 4.0.0 Resolution: Fixed > Fix HIVE_COMPACTOR_COMPACT_MM property > -- > > Key: HIVE-24785 > URL: https://issues.apache.org/jira/browse/HIVE-24785 > Project: Hive > Issue Type: Bug >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently it will disable query based compaction for mm tables, but the > Worker will fall back to MR based compaction which is not implemented for mm > tables. > This property should disable compaction in the Initiator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24785) Fix HIVE_COMPACTOR_COMPACT_MM property
[ https://issues.apache.org/jira/browse/HIVE-24785?focusedWorklogId=554644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554644 ] ASF GitHub Bot logged work on HIVE-24785: - Author: ASF GitHub Bot Created on: 19/Feb/21 08:22 Start Date: 19/Feb/21 08:22 Worklog Time Spent: 10m Work Description: pvargacl merged pull request #1979: URL: https://github.com/apache/hive/pull/1979 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554644) Time Spent: 2.5h (was: 2h 20m) > Fix HIVE_COMPACTOR_COMPACT_MM property > -- > > Key: HIVE-24785 > URL: https://issues.apache.org/jira/browse/HIVE-24785 > Project: Hive > Issue Type: Bug >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently it will disable query based compaction for mm tables, but the > Worker will fall back to MR based compaction which is not implemented for mm > tables. > This property should disable compaction in the Initiator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24772) Revamp Server Request Error Logging
[
https://issues.apache.org/jira/browse/HIVE-24772?focusedWorklogId=554641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554641
]
ASF GitHub Bot logged work on HIVE-24772:
-
Author: ASF GitHub Bot
Created on: 19/Feb/21 08:20
Start Date: 19/Feb/21 08:20
Worklog Time Spent: 10m
Work Description: zchovan commented on a change in pull request #1974:
URL: https://github.com/apache/hive/pull/1974#discussion_r578999876
##
File path:
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
##
@@ -302,7 +301,7 @@ public TRenewDelegationTokenResp
RenewDelegationToken(TRenewDelegationTokenReq r
hiveAuthFactory, req.getDelegationToken());
resp.setStatus(OK_STATUS);
} catch (HiveSQLException e) {
-LOG.error("Error obtaining renewing token", e);
+LOG.error("Failed to renewing token [request: {}]", e);
Review comment:
nit: "Failed to renew" instead of "Failed to renewing"
##
File path:
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
##
@@ -283,7 +282,7 @@ public TCancelDelegationTokenResp
CancelDelegationToken(TCancelDelegationTokenRe
hiveAuthFactory, req.getDelegationToken());
resp.setStatus(OK_STATUS);
} catch (HiveSQLException e) {
-LOG.error("Error canceling delegation token", e);
+LOG.error("Failed to canceling delegation token [request: {}]", req,
e);
Review comment:
nit: "Failed to cancel" instead of "Failed to canceling"
##
File path:
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
##
@@ -552,7 +554,7 @@ public TGetInfoResp GetInfo(TGetInfoReq req) throws
TException {
resp.setInfoValue(getInfoValue.toTGetInfoValue());
resp.setStatus(OK_STATUS);
} catch (Exception e) {
- LOG.warn("Error getting info: ", e);
+ LOG.warn("Error getting info", e);
Review comment:
should this be error level log? or "Failed to get info"
##
File path:
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java
##
@@ -536,7 +538,7 @@ public TCloseSessionResp CloseSession(TCloseSessionReq req)
throws TException {
context.setSessionHandle(null);
}
} catch (Exception e) {
- LOG.warn("Error closing session: ", e);
+ LOG.warn("Error closing session", e);
Review comment:
should this be error level log? or maybe change it to "Failed to close
the session"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 554641)
Time Spent: 1h (was: 50m)
> Revamp Server Request Error Logging
> ---
>
> Key: HIVE-24772
> URL: https://issues.apache.org/jira/browse/HIVE-24772
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Most of the action takes place in {{ThriftCLIService}} where errors are
> logged in response to client requests (though I know in many instances things
> are logged multiple times).
> I propose to improve this on multiple fronts:
> # Many log messages have the word "Error" in it, but log at the WARN level.
> I have changed all relevant logging to be at ERROR level and removed the word
> "Error" from the message
> # Some of the error message in the logging code had copy & paste errors where
> they printed the wrong request name
> # Print the actual request object in the error message
> # Big one for me: Do not pass a stack trace to the client. This is bad
> practice from a security perspective,... clients should not know that level
> of detail of the server, and also it's very confusing for the client
> perspective to understand that the stack trace is actually from the remote
> server, not the local client, and finally, it's too messy for a typical user
> to deal with anyway. Stack trace should be presented in the HS2 logs only.
> # Various clean up
> # Log an IP address for the client as part of standard operating procedure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
