[jira] [Updated] (HIVE-13648) ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR, or DECIMAL when maxLength or precision/scale is different
[
https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13648:
Status: In Progress (was: Patch Available)
> ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR,
> or DECIMAL when maxLength or precision/scale is different
> --
>
> Key: HIVE-13648
> URL: https://issues.apache.org/jira/browse/HIVE-13648
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13648.01.patch, HIVE-13648.02.patch,
> HIVE-13648.03.patch
>
>
> E.g. when a data file is copied in has a VARCHAR maxLength that doesn't match
> the DDL's maxLength. This error is produced:
> {code}
> java.io.IOException: ORC does not support type conversion from file type
> varchar(145) (36) to reader type varchar(114) (36)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13648) ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR, or DECIMAL when maxLength or precision/scale is different
[
https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13648:
Status: Patch Available (was: In Progress)
> ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR,
> or DECIMAL when maxLength or precision/scale is different
> --
>
> Key: HIVE-13648
> URL: https://issues.apache.org/jira/browse/HIVE-13648
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13648.01.patch, HIVE-13648.02.patch,
> HIVE-13648.03.patch
>
>
> E.g. when a data file is copied in has a VARCHAR maxLength that doesn't match
> the DDL's maxLength. This error is produced:
> {code}
> java.io.IOException: ORC does not support type conversion from file type
> varchar(145) (36) to reader type varchar(114) (36)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13648) ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR, or DECIMAL when maxLength or precision/scale is different
[
https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13648:
Attachment: HIVE-13648.03.patch
> ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR,
> or DECIMAL when maxLength or precision/scale is different
> --
>
> Key: HIVE-13648
> URL: https://issues.apache.org/jira/browse/HIVE-13648
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13648.01.patch, HIVE-13648.02.patch,
> HIVE-13648.03.patch
>
>
> E.g. when a data file is copied in has a VARCHAR maxLength that doesn't match
> the DDL's maxLength. This error is produced:
> {code}
> java.io.IOException: ORC does not support type conversion from file type
> varchar(145) (36) to reader type varchar(114) (36)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: Patch Available (was: In Progress) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.04.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: In Progress (was: Patch Available) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13590) Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
[
https://issues.apache.org/jira/browse/HIVE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332931#comment-15332931
]
Chaoyu Tang commented on HIVE-13590:
The patch has been uploaded to https://reviews.apache.org/r/48771/ for review.
> Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
> -
>
> Key: HIVE-13590
> URL: https://issues.apache.org/jira/browse/HIVE-13590
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Security
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13590.patch
>
>
> In a kerberized HS2 with LDAP authentication enabled, LDAP user usually logs
> in using username in form of username@domain in LDAP multi-domain case. But
> it fails if the domain was not in the Hadoop auth_to_local mapping rule, the
> error is as following:
> {code}
> Caused by:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to ct...@mydomain.com
> at
> org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
> at org.apache.hadoop.security.User.(User.java:48)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13590) Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
[
https://issues.apache.org/jira/browse/HIVE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-13590:
---
Status: Patch Available (was: Open)
> Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
> -
>
> Key: HIVE-13590
> URL: https://issues.apache.org/jira/browse/HIVE-13590
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Security
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13590.patch
>
>
> In a kerberized HS2 with LDAP authentication enabled, LDAP user usually logs
> in using username in form of username@domain in LDAP multi-domain case. But
> it fails if the domain was not in the Hadoop auth_to_local mapping rule, the
> error is as following:
> {code}
> Caused by:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to ct...@mydomain.com
> at
> org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
> at org.apache.hadoop.security.User.(User.java:48)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13590) Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
[
https://issues.apache.org/jira/browse/HIVE-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-13590:
---
Attachment: HIVE-13590.patch
[~szehon], [~spena] could you help review the patch? Thanks.
> Kerberized HS2 with LDAP auth enabled fails in multi-domain LDAP case
> -
>
> Key: HIVE-13590
> URL: https://issues.apache.org/jira/browse/HIVE-13590
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Security
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13590.patch
>
>
> In a kerberized HS2 with LDAP authentication enabled, LDAP user usually logs
> in using username in form of username@domain in LDAP multi-domain case. But
> it fails if the domain was not in the Hadoop auth_to_local mapping rule, the
> error is as following:
> {code}
> Caused by:
> org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule:
> No rules applied to ct...@mydomain.com
> at
> org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
> at org.apache.hadoop.security.User.(User.java:48)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Summary: Update Spark version to 2.0.0 (was: Update Spark version to 1.6) > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 1.6
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332903#comment-15332903 ] Ferdinand Xu commented on HIVE-14029: - cc [~xuefuz] > Update Spark version to 1.6 > --- > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13976) UNION ALL which takes actual source table in one side failed
[
https://issues.apache.org/jira/browse/HIVE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332899#comment-15332899
]
Amir Shenavandeh commented on HIVE-13976:
-
This seems to be fixed in HIVE-2 and above.
> UNION ALL which takes actual source table in one side failed
>
>
> Key: HIVE-13976
> URL: https://issues.apache.org/jira/browse/HIVE-13976
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
> Environment: Ubuntu 12.04, JDK 7
>Reporter: Kai Sasaki
>
> UNION ALL must take actual source table in both side or none exclusively.
> * UNION ALL with actual table in both side -> Succeed as expected
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> FROM
> table1
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> FROM
> table2
> {code}
> * UNION ALL without actual table in both side -> Succeed as expected
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> {code}
> * UNION ALL with actual table on one side -> Failed
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> FROM
>some_table
> {code}
> The error message from map task of third case is this.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.IllegalArgumentException: Can not create a Path from an
> empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
> at org.apache.hadoop.fs.Path.(Path.java:135)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.getPath(HiveInputFormat.java:116)
> at org.apache.hadoop.mapred.MapTask.updateJobWithSplit(MapTask.java:458)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13976) UNION ALL which takes actual source table in one side failed
[
https://issues.apache.org/jira/browse/HIVE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332899#comment-15332899
]
Amir Shenavandeh edited comment on HIVE-13976 at 6/16/16 2:10 AM:
--
This seems to be fixed in HIVE-2 and above. Still interesting to see which
patch has fixed it.
was (Author: shenavandeh):
This seems to be fixed in HIVE-2 and above.
> UNION ALL which takes actual source table in one side failed
>
>
> Key: HIVE-13976
> URL: https://issues.apache.org/jira/browse/HIVE-13976
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
> Environment: Ubuntu 12.04, JDK 7
>Reporter: Kai Sasaki
>
> UNION ALL must take actual source table in both side or none exclusively.
> * UNION ALL with actual table in both side -> Succeed as expected
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> FROM
> table1
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> FROM
> table2
> {code}
> * UNION ALL without actual table in both side -> Succeed as expected
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> {code}
> * UNION ALL with actual table on one side -> Failed
> {code}
> SELECT
> 1 AS id,
> 'Alice' AS name
> UNION ALL
> SELECT
> 2 AS id,
> 'Bob' AS name
> FROM
>some_table
> {code}
> The error message from map task of third case is this.
> {code}
> Diagnostic Messages for this Task:
> Error: java.lang.IllegalArgumentException: Can not create a Path from an
> empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
> at org.apache.hadoop.fs.Path.(Path.java:135)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.getPath(HiveInputFormat.java:116)
> at org.apache.hadoop.mapred.MapTask.updateJobWithSplit(MapTask.java:458)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng reassigned HIVE-13934: Assignee: Wei Zheng (was: Siddharth Seth) > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14025) Insert overwrite does not work in HBase tables
[ https://issues.apache.org/jira/browse/HIVE-14025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14025: --- Description: This problem sounds serious and I have filed two JIRAs for them. The first one is a correctness issue. We inserted 500 rows but got only one row. The second one is related to stats. There is no move task when we insert into a Hbase table. Then there is no stats task as well. As a result, we continue to assume that the stats is accurate. > Insert overwrite does not work in HBase tables > -- > > Key: HIVE-14025 > URL: https://issues.apache.org/jira/browse/HIVE-14025 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > This problem sounds serious and I have filed two JIRAs for them. The first > one is a correctness issue. We inserted 500 rows but got only one row. The > second one is related to stats. There is no move task when we insert into a > Hbase table. Then there is no stats task as well. As a result, we continue to > assume that the stats is accurate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14026) data can not be retrieved
[
https://issues.apache.org/jira/browse/HIVE-14026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332787#comment-15332787
]
Pengcheng Xiong commented on HIVE-14026:
[~ashutoshc], do you think we should ask someone from HBase to take a look or
we should take a look? Thanks.
> data can not be retrieved
> -
>
> Key: HIVE-14026
> URL: https://issues.apache.org/jira/browse/HIVE-14026
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>
> {code}
> DROP TABLE users;
> CREATE TABLE users(key string, state string, country string, country_id int)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> INSERT OVERWRITE TABLE users SELECT 'user1', 'IA', 'USA', 0 FROM src;
> select * from users;
> {code}
> The result is only one row:
> {code}
> user1 IA USA 0
> {code}
> should be 500 rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332778#comment-15332778
]
Yongzhi Chen commented on HIVE-14006:
-
LGTM +1
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332774#comment-15332774
]
Hive QA commented on HIVE-13884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810920/HIVE-13884.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 217 failed/errored test(s), 10233 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucketpruning1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_windowing
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_4
[jira] [Commented] (HIVE-14024) setAllColumns is called incorrectly after some changes
[ https://issues.apache.org/jira/browse/HIVE-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332734#comment-15332734 ] Prasanth Jayachandran commented on HIVE-14024: -- LGTM, +1 > setAllColumns is called incorrectly after some changes > -- > > Key: HIVE-14024 > URL: https://issues.apache.org/jira/browse/HIVE-14024 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14024.patch > > > h/t [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332731#comment-15332731
]
Naveen Gangam commented on HIVE-14006:
--
The test failures do not appear to be related to the fix.
[~ychena] [~aihuaxu] can you please review the patch? Thanks
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. I ran some tests locally, HiveQA repeatedly fails to run due to JIRA instability. > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-13771.01.patch, HIVE-13771.02.patch, > HIVE-13771.03.patch, HIVE-13771.04.patch, HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14024) setAllColumns is called incorrectly after some changes
[ https://issues.apache.org/jira/browse/HIVE-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14024: Status: Patch Available (was: Open) > setAllColumns is called incorrectly after some changes > -- > > Key: HIVE-14024 > URL: https://issues.apache.org/jira/browse/HIVE-14024 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14024.patch > > > h/t [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14024) setAllColumns is called incorrectly after some changes
[ https://issues.apache.org/jira/browse/HIVE-14024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14024: Attachment: HIVE-14024.patch [~prasanth_j] can you take a look? The blank actually means read no columns > setAllColumns is called incorrectly after some changes > -- > > Key: HIVE-14024 > URL: https://issues.apache.org/jira/browse/HIVE-14024 > Project: Hive > Issue Type: Bug >Reporter: Takahiko Saito >Assignee: Sergey Shelukhin > Attachments: HIVE-14024.patch > > > h/t [~gopalv] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: Patch Available (was: In Progress) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: In Progress (was: Patch Available) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.03.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13771) LLAPIF: generate app ID
[ https://issues.apache.org/jira/browse/HIVE-13771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13771: Attachment: HIVE-13771.04.patch And again > LLAPIF: generate app ID > --- > > Key: HIVE-13771 > URL: https://issues.apache.org/jira/browse/HIVE-13771 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13771.01.patch, HIVE-13771.02.patch, > HIVE-13771.03.patch, HIVE-13771.04.patch, HIVE-13771.patch > > > See comments in the HIVE-13675 patch. The uniqueness needs to be ensured; the > user may be allowed to supply a prefix (e.g. his YARN app Id, if any) for > ease of tracking -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13964) Add a parameter to beeline to allow a properties file to be passed in
[ https://issues.apache.org/jira/browse/HIVE-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abdullah Yousufi updated HIVE-13964: Attachment: HIVE-13964.05.patch > Add a parameter to beeline to allow a properties file to be passed in > - > > Key: HIVE-13964 > URL: https://issues.apache.org/jira/browse/HIVE-13964 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 2.0.1 >Reporter: Abdullah Yousufi >Assignee: Abdullah Yousufi >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-13964.01.patch, HIVE-13964.02.patch, > HIVE-13964.03.patch, HIVE-13964.04.patch, HIVE-13964.05.patch > > > HIVE-6652 removed the ability to pass in a properties file as a beeline > parameter. It may be a useful feature to be able to pass the file in is a > parameter, such as --property-file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Fix Version/s: 2.1.0 > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Priority: Blocker (was: Critical) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13884:
---
Attachment: HIVE-13884.4.patch
> Disallow queries fetching more than a configured number of partitions in
> PartitionPruner
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch,
> HIVE-13884.3.patch, HIVE-13884.4.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332514#comment-15332514 ] Sergey Shelukhin commented on HIVE-13974: - +1 pending tests and a resolution of [~ekoifman]'s last comment. > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332510#comment-15332510
]
Hive QA commented on HIVE-14006:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810883/HIVE-14006.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10233 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/127/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/127/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-127/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810883 - PreCommit-HIVE-MASTER-Build
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13984) Use multi-threaded approach to listing files for msck
[ https://issues.apache.org/jira/browse/HIVE-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332412#comment-15332412 ] Prasanth Jayachandran commented on HIVE-13984: -- LGTM, +1 > Use multi-threaded approach to listing files for msck > - > > Key: HIVE-13984 > URL: https://issues.apache.org/jira/browse/HIVE-13984 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13984.01.patch, HIVE-13984.02.patch, > HIVE-13984.03.patch, HIVE-13984.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[
https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332304#comment-15332304
]
Eugene Koifman commented on HIVE-13974:
---
[~mmccline], here is the complete ACID test.
The version currently included in the patch is only to help repro the issue.
That test won't fail regardless if compaction succeeds.
One below will.
{noformat}
@Test
public void testCompactWithDelete() throws Exception {
int[][] tableData = {{1,2},{3,4}};
runStatementOnDriver("insert into " + Table.ACIDTBL + "(a,b) " +
makeValuesClause(tableData));
runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MAJOR'");
Worker t = new Worker();
t.setThreadId((int) t.getId());
t.setHiveConf(hiveConf);
AtomicBoolean stop = new AtomicBoolean();
AtomicBoolean looped = new AtomicBoolean();
stop.set(true);
t.init(stop, looped);
t.run();
runStatementOnDriver("delete from " + Table.ACIDTBL + " where b = 4");
runStatementOnDriver("update " + Table.ACIDTBL + " set b = -2 where b = 2");
runStatementOnDriver("alter table "+ Table.ACIDTBL + " compact 'MINOR'");
t.run();
TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
ShowCompactResponse resp = txnHandler.showCompact(new ShowCompactRequest());
Assert.assertEquals("Unexpected number of compactions in history", 2,
resp.getCompactsSize());
Assert.assertEquals("Unexpected 0 compaction state",
TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(0).getState());
Assert.assertEquals("Unexpected 1 compaction state",
TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(1).getState());
}
{noformat}
> ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
> ---
>
> Key: HIVE-13974
> URL: https://issues.apache.org/jira/browse/HIVE-13974
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC, Transactions
>Affects Versions: 1.3.0, 2.1.0, 2.2.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch
>
>
> Currently, the included columns are based on the fileSchema and not the
> readerSchema which doesn't work for adding columns to non-last STRUCT data
> type columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14020:
---
Fix Version/s: (was: 2.1.0)
2.1.1
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14020.1.patch
>
>
> NO PRECOMMIT TESTS
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Fix Version/s: 2.2.0
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0, 2.2.0
>
> Attachments: HIVE-14020.1.patch
>
>
> NO PRECOMMIT TESTS
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-14020.1.patch
>
>
> NO PRECOMMIT TESTS
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Description:
NO PRECOMMIT TESTS
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
+-+--+
| Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
+-+--+
1 row selected (0.072 seconds)
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
Error: ORA-00922: missing or invalid option (state=42000,code=922)
Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
... 8 more
*** schemaTool failed ***
{noformat}
At the face of it, it looks like issue from the actual script (
034-HIVE-13076.oracle.sql ) that's provided:
{noformat}
CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
(
CHILD_CD_ID NUMBER,
CHILD_INTEGER_IDX NUMBER,
CHILD_TBL_ID NUMBER,
PARENT_CD_ID NUMBER NOT NULL,
PARENT_INTEGER_IDX NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL,
POSITION NUMBER NOT NULL,
CONSTRAINT_NAME VARCHAR(400) NOT NULL,
CONSTRAINT_TYPE NUMBER NOT NULL,
UPDATE_RULE NUMBER,
DELETE_RULE NUMBER,
ENABLE_VALIDATE_RELY NUMBER NOT NULL
) ;
ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
(CONSTRAINT_NAME, POSITION);
CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
{noformat}
was:
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS
[jira] [Commented] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332257#comment-15332257
]
Sushanth Sowmyan commented on HIVE-14020:
-
+1
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14020.1.patch
>
>
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Attachment: HIVE-14020.1.patch
[~sushanth]] Can you please review this.
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14020.1.patch
>
>
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Status: Patch Available (was: Open)
> Hive MS restart failed during EU with ORA-00922 error as part of DB schema
> upgrade
> --
>
> Key: HIVE-14020
> URL: https://issues.apache.org/jira/browse/HIVE-14020
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14020.1.patch
>
>
> The underlying failure seems to be visible from --verbose :
> {noformat}
> Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
> Metastore connection User: hiveuser
> Starting upgrade metastore schema from version 2.0.0 to 2.1.0
> Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
> Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> Connected to: Oracle (version Oracle Database 11g Express Edition Release
> 11.2.0.2.0 - 64bit Production)
> Driver: Oracle JDBC driver (version 11.2.0.4.0)
> Transaction isolation: TRANSACTION_READ_COMMITTED
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
> Autocommit status: true
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore
> schema from 2.0.0 to 2.1.0' AS Status from dual
> +-+--+
> | STATUS |
> +-+--+
> | Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
> +-+--+
> 1 row selected (0.072 seconds)
> 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
> KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
> NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
> VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
> Error: ORA-00922: missing or invalid option (state=42000,code=922)
> Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> Underlying cause: java.io.IOException : Schema script failed, errorcode 2
> org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
> state would be inconsistent !!
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
> at
> org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Schema script failed, errorcode 2
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
> at
> org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
> at
> org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
> ... 8 more
> *** schemaTool failed ***
> {noformat}
> At the face of it, it looks like issue from the actual script (
> 034-HIVE-13076.oracle.sql ) that's provided:
> {noformat}
> CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
> (
> CHILD_CD_ID NUMBER,
> CHILD_INTEGER_IDX NUMBER,
> CHILD_TBL_ID NUMBER,
> PARENT_CD_ID NUMBER NOT NULL,
> PARENT_INTEGER_IDX NUMBER NOT NULL,
> PARENT_TBL_ID NUMBER NOT NULL,
> POSITION NUMBER NOT NULL,
> CONSTRAINT_NAME VARCHAR(400) NOT NULL,
> CONSTRAINT_TYPE NUMBER NOT NULL,
> UPDATE_RULE NUMBER,
> DELETE_RULE NUMBER,
> ENABLE_VALIDATE_RELY NUMBER NOT NULL
> ) ;
> ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
> (CONSTRAINT_NAME, POSITION);
> CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332238#comment-15332238 ] Nachiket Vaidya commented on HIVE-13966: [~sircodesalot]: Thank you for your inputs. By "synchronous" I mean running notifications in the same transaction so that if notification fails entire operation fails. As Sravya pointed out that "synchronous" may be misnomer (may be name could be same transaction notifier) I do not know what is the "contract" for metastore listener - if we are supposed to run them after operation is committed/rolled back. We can use the same class hierarchy and will have different config property. > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Priority: Critical > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Description:
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
+-+--+
| Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
+-+--+
1 row selected (0.072 seconds)
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
Error: ORA-00922: missing or invalid option (state=42000,code=922)
Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
... 8 more
*** schemaTool failed ***
{noformat}
At the face of it, it looks like issue from the actual script (
034-HIVE-13076.oracle.sql ) that's provided:
{noformat}
CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
(
CHILD_CD_ID NUMBER,
CHILD_INTEGER_IDX NUMBER,
CHILD_TBL_ID NUMBER,
PARENT_CD_ID NUMBER NOT NULL,
PARENT_INTEGER_IDX NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL,
POSITION NUMBER NOT NULL,
CONSTRAINT_NAME VARCHAR(400) NOT NULL,
CONSTRAINT_TYPE NUMBER NOT NULL,
UPDATE_RULE NUMBER,
DELETE_RULE NUMBER,
ENABLE_VALIDATE_RELY NUMBER NOT NULL
) ;
ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
(CONSTRAINT_NAME, POSITION);
CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
{noformat}
was:
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
[jira] [Updated] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
[
https://issues.apache.org/jira/browse/HIVE-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-14020:
-
Description:
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
+-+--+
| Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
+-+--+
1 row selected (0.072 seconds)
0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE> CREATE TABLE IF NOT EXISTS
KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
Error: ORA-00922: missing or invalid option (state=42000,code=922)
Closing: 0: jdbc:oracle:thin:@//aaa:bb:cc:dd:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
... 8 more
*** schemaTool failed ***
{noformat}
At the face of it, it looks like bad ^Ms got added from the actual script (
034-HIVE-13076.oracle.sql ) that's provided:
{noformat}
CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
(
CHILD_CD_ID NUMBER,
CHILD_INTEGER_IDX NUMBER,
CHILD_TBL_ID NUMBER,
PARENT_CD_ID NUMBER NOT NULL,
PARENT_INTEGER_IDX NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL,
POSITION NUMBER NOT NULL,
CONSTRAINT_NAME VARCHAR(400) NOT NULL,
CONSTRAINT_TYPE NUMBER NOT NULL,
UPDATE_RULE NUMBER,
DELETE_RULE NUMBER,
ENABLE_VALIDATE_RELY NUMBER NOT NULL
) ;
ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
(CONSTRAINT_NAME, POSITION);
CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
{noformat}
was:
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//172.22.66.99:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.66.99:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.66.99:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.66.99:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
[jira] [Assigned] (HIVE-11089) Hive Streaming: connection fails when using a proxy user UGI
[
https://issues.apache.org/jira/browse/HIVE-11089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng reassigned HIVE-11089:
Assignee: Wei Zheng
> Hive Streaming: connection fails when using a proxy user UGI
>
>
> Key: HIVE-11089
> URL: https://issues.apache.org/jira/browse/HIVE-11089
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.14.0, 1.0.0, 1.2.0
>Reporter: Adam Kunicki
>Assignee: Wei Zheng
> Labels: ACID, Streaming
>
> HIVE-7508 "Add Kerberos Support" seems to also remove the ability to specify
> a proxy user.
> HIVE-8427 adds a call to ugi.hasKerberosCredentials() to check whether the
> connection is supposed to be a secure connection.
> This however breaks support for Proxy Users as a proxy user UGI will always
> return false to hasKerberosCredentials().
> See lines 273, 274 of HiveEndPoint.java
> {code}
> this.secureMode = ugi==null ? false : ugi.hasKerberosCredentials();
> this.msClient = getMetaStoreClient(endPoint, conf, secureMode);
> {code}
> It also seems that between 13.1 and 0.14 the newConnection() method that
> includes a proxy user has been removed.
> for reference:
> https://github.com/apache/hive/commit/8e423a12db47759196c24535fbc32236b79f464a
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14003) queries running against llap hang at times - preemption issues
[ https://issues.apache.org/jira/browse/HIVE-14003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332229#comment-15332229 ] Siddharth Seth commented on HIVE-14003: --- [~sershe], [~prasanth_j] - is the patch good to go in (with the comments changed to point to new jiras which will be created). I think there are additional cases which need to be addressed; they can be addressed in a different jira. This one, in it's current form, does get rid of queries getting stuck on hash table creation. > queries running against llap hang at times - preemption issues > -- > > Key: HIVE-14003 > URL: https://issues.apache.org/jira/browse/HIVE-14003 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.1.0 >Reporter: Takahiko Saito >Assignee: Siddharth Seth > Attachments: HIVE-14003.01.patch > > > The preemption logic in the Hive processor needs some more work. There are > definitely windows where the abort flag is completely dropped within the Hive > processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13956) LLAP: external client output is writing to channel before it is writable again
[ https://issues.apache.org/jira/browse/HIVE-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13956: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) committed to master > LLAP: external client output is writing to channel before it is writable again > -- > > Key: HIVE-13956 > URL: https://issues.apache.org/jira/browse/HIVE-13956 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.2.0 > > Attachments: HIVE-13956.1.patch, HIVE-13956.2.patch > > > Rows are being written/flushed on the output channel without checking if the > channel is writable. Introduce a writability check/wait. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332208#comment-15332208 ] Ashutosh Chauhan commented on HIVE-14018: - +1 > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13759) LlapTaskUmbilicalExternalClient should be closed by the record reader
[ https://issues.apache.org/jira/browse/HIVE-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13759: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > LlapTaskUmbilicalExternalClient should be closed by the record reader > - > > Key: HIVE-13759 > URL: https://issues.apache.org/jira/browse/HIVE-13759 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.2.0 > > Attachments: HIVE-13759.1.patch, HIVE-13759.2.patch, > HIVE-13759.3.patch > > > The umbilical external client (and the server socket it creates) doesn't look > like it's getting closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13759) LlapTaskUmbilicalExternalClient should be closed by the record reader
[ https://issues.apache.org/jira/browse/HIVE-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13759: -- Attachment: HIVE-13759.3.patch rebasing > LlapTaskUmbilicalExternalClient should be closed by the record reader > - > > Key: HIVE-13759 > URL: https://issues.apache.org/jira/browse/HIVE-13759 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13759.1.patch, HIVE-13759.2.patch, > HIVE-13759.3.patch > > > The umbilical external client (and the server socket it creates) doesn't look > like it's getting closed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[ https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332177#comment-15332177 ] Szehon Ho commented on HIVE-14015: -- Ah we did not handle the kerberized case. Thanks very much for fixing this! > SMB MapJoin failed for Hive on Spark when kerberized > > > Key: HIVE-14015 > URL: https://issues.apache.org/jira/browse/HIVE-14015 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.1.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14015.1.patch > > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can be issued only with kerberos or web authentication > It could be reproduced: > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.execution.engine=spark; > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13833) Add an initial delay when starting the heartbeat
[ https://issues.apache.org/jira/browse/HIVE-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332141#comment-15332141 ] Wei Zheng commented on HIVE-13833: -- Also committed to branch-2.1 > Add an initial delay when starting the heartbeat > > > Key: HIVE-13833 > URL: https://issues.apache.org/jira/browse/HIVE-13833 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Minor > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13833.1.patch, HIVE-13833.2.patch, > HIVE-13833.3.patch, HIVE-13833.4.patch > > > Since the scheduling of heartbeat happens immediately after lock acquisition, > it's unnecessary to send heartbeat at the time when locks is acquired. Add an > initial delay to skip this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13833) Add an initial delay when starting the heartbeat
[ https://issues.apache.org/jira/browse/HIVE-13833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13833: - Fix Version/s: 2.1.1 > Add an initial delay when starting the heartbeat > > > Key: HIVE-13833 > URL: https://issues.apache.org/jira/browse/HIVE-13833 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0, 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Minor > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13833.1.patch, HIVE-13833.2.patch, > HIVE-13833.3.patch, HIVE-13833.4.patch > > > Since the scheduling of heartbeat happens immediately after lock acquisition, > it's unnecessary to send heartbeat at the time when locks is acquired. Add an > initial delay to skip this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13961) ACID: Major compaction fails to include the original bucket files if there's no delta directory
[ https://issues.apache.org/jira/browse/HIVE-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13961: - Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 1.3.0 Status: Resolved (was: Patch Available) Thanks Eugene for the review. Committed to master, branch-2.1, and branch-1. > ACID: Major compaction fails to include the original bucket files if there's > no delta directory > --- > > Key: HIVE-13961 > URL: https://issues.apache.org/jira/browse/HIVE-13961 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng >Priority: Blocker > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-13961.1.patch, HIVE-13961.2.patch, > HIVE-13961.3.patch, HIVE-13961.4.patch, HIVE-13961.5.patch, HIVE-13961.6.patch > > > The issue can be reproduced by steps below: > 1. Insert a row to Non-ACID table > 2. Convert Non-ACID to ACID table (i.e. set transactional=true table property) > 3. Perform Major compaction -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both
[ https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332125#comment-15332125 ] Sergio Peña commented on HIVE-14001: Thanks [~pvary] .The patch looks good +1 > beeline doesn't give out an error when takes either "-e" or "-f" in command > instead of both > --- > > Key: HIVE-14001 > URL: https://issues.apache.org/jira/browse/HIVE-14001 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 0.10.0, 2.0.1 >Reporter: Peter Vary >Priority: Trivial > Attachments: HIVE-14001.patch > > > When providing both arguments there should be an error message -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14006:
-
Status: Open (was: Patch Available)
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14006:
-
Status: Patch Available (was: Open)
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14006:
-
Attachment: HIVE-14006.1.patch
Adding a testcase in this version of the patch.
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.1.patch, HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332072#comment-15332072 ] Matt McCline commented on HIVE-13974: - [~sershe] Thank you Sergey for the review. I added comments for the items and also added a unit test case that Eugene found was failing in ACID that is fixed by this change. > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.02.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: Patch Available (was: In Progress) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: In Progress (was: Patch Available) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13974.01.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14006:
-
Status: Patch Available (was: Open)
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14006) Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14006:
-
Attachment: HIVE-14006.patch
> Hive query with UNION ALL fails with ArrayIndexOutOfBoundsException
> ---
>
> Key: HIVE-14006
> URL: https://issues.apache.org/jira/browse/HIVE-14006
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14006.patch
>
>
> set hive.cbo.enable=false;
> DROP VIEW IF EXISTS a_view;
> DROP TABLE IF EXISTS table_a1;
> DROP TABLE IF EXISTS table_a2;
> DROP TABLE IF EXISTS table_b1;
> DROP TABLE IF EXISTS table_b2;
> CREATE TABLE table_a1
> (composite_key STRING);
> CREATE TABLE table_a2
> (composite_key STRING);
> CREATE TABLE table_b1
> (composite_key STRING, col1 STRING);
> CREATE TABLE table_b2
> (composite_key STRING);
> CREATE VIEW a_view AS
> SELECT
> substring(a1.composite_key, 1, locate('|',a1.composite_key) - 1) AS autoname,
> NULL AS col1
> FROM table_a1 a1
> FULL OUTER JOIN table_a2 a2
> ON a1.composite_key = a2.composite_key
> UNION ALL
> SELECT
> substring(b1.composite_key, 1, locate('|',b1.composite_key) - 1) AS autoname,
> b1.col1 AS col1
> FROM table_b1 b1
> FULL OUTER JOIN table_b2 b2
> ON b1.composite_key = b2.composite_key;
> INSERT INTO TABLE table_b1
> SELECT * FROM (
> SELECT 'something|awful', 'col1'
> )s ;
> SELECT autoname
> FROM a_view
> WHERE autoname='something';
> fails with
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"_col0":"something"}
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"_col0":"something"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
> ... 8 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:134)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
> The same query succeeds when {{hive.ppd.remove.duplicatefilters=false}} with
> or without CBO on. It also succeeds with just CBO on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13987) Clarify current error shown when HS2 is down
[
https://issues.apache.org/jira/browse/HIVE-13987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-13987:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~ayousufi] for your contribution. I committed this to master.
> Clarify current error shown when HS2 is down
>
>
> Key: HIVE-13987
> URL: https://issues.apache.org/jira/browse/HIVE-13987
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.1
>Reporter: Abdullah Yousufi
>Assignee: Abdullah Yousufi
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-13987.01.patch, HIVE-13987.02.patch
>
>
> When HS2 is down and a query is run, the following error is shown in beeline:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
> It may be more helpful to also indicate that the reason for this is that HS2
> is down, such as:
> {code}
> 0: jdbc:hive2://localhost:1> show tables;
> HS2 may be unavailable, check server status
> Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14018: --- Attachment: HIVE-14018.patch > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14018: --- Status: Patch Available (was: In Progress) > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both
[ https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332010#comment-15332010 ] Peter Vary commented on HIVE-14001: --- I think these issues are unrelated, and I have checked the other QA runs, and I see these errors there too. Could someone please review my patch? Thanks, Peter > beeline doesn't give out an error when takes either "-e" or "-f" in command > instead of both > --- > > Key: HIVE-14001 > URL: https://issues.apache.org/jira/browse/HIVE-14001 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 0.10.0, 2.0.1 >Reporter: Peter Vary >Priority: Trivial > Attachments: HIVE-14001.patch > > > When providing both arguments there should be an error message -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14018) Make IN clause row selectivity estimation customizable
[ https://issues.apache.org/jira/browse/HIVE-14018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14018 started by Jesus Camacho Rodriguez. -- > Make IN clause row selectivity estimation customizable > -- > > Key: HIVE-14018 > URL: https://issues.apache.org/jira/browse/HIVE-14018 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 2.1.0, 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-14018.patch > > > After HIVE-13287 went in, we calculate IN clause estimates natively (instead > of just dividing incoming number of rows by 2). However, as the distribution > of values of the columns is considered uniform, we might end up heavily > underestimating/overestimating the resulting number of rows. > This issue is to add a factor that multiplies the IN clause estimation so we > can alleviate this problem. The solution is not very elegant, but it is the > best we can do until we have histograms to improve our estimate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[ https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331989#comment-15331989 ] Chaoyu Tang commented on HIVE-14015: LGTM, +1 pending on testing. > SMB MapJoin failed for Hive on Spark when kerberized > > > Key: HIVE-14015 > URL: https://issues.apache.org/jira/browse/HIVE-14015 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.1.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14015.1.patch > > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can be issued only with kerberos or web authentication > It could be reproduced: > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.execution.engine=spark; > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14001) beeline doesn't give out an error when takes either "-e" or "-f" in command instead of both
[
https://issues.apache.org/jira/browse/HIVE-14001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331979#comment-15331979
]
Peter Vary commented on HIVE-14001:
---
The test are run, but could not be published because of a JIRA problem - here
are the results:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809859/HIVE-14001.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10226 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/108/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/108/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-108/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
> beeline doesn't give out an error when takes either "-e" or "-f" in command
> instead of both
> ---
>
> Key: HIVE-14001
> URL: https://issues.apache.org/jira/browse/HIVE-14001
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 0.10.0, 2.0.1
>Reporter: Peter Vary
>Priority: Trivial
> Attachments: HIVE-14001.patch
>
>
> When providing both arguments there should be an error message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14009) Acid DB creation error in HiveQA
[
https://issues.apache.org/jira/browse/HIVE-14009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331901#comment-15331901
]
Sergio Peña commented on HIVE-14009:
That's weird. TestEncryptdHDFSCliDriver is just a clone of TestCliDriver that
prepares encryption during the tests. But all tests are run sequentially as
other tests do.
I found there is one test that runs queries in parallel, I don't know if it has
something to do with this. Test is {{TestMTQueries}}
> Acid DB creation error in HiveQA
>
>
> Key: HIVE-14009
> URL: https://issues.apache.org/jira/browse/HIVE-14009
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Seen when running TestEncryptedHDFSCliDriver, at least with Hadoop 2.7.2
> (HIVE-13930).
> Looks like such issues are usually caused by concurrent db creation from
> multiple threads.
> {noformat}
> java.lang.RuntimeException: Unable to set up transaction database for
> testing: Exception during creation of file
> /home/hiveptest/54.219.24.101-hiveptest-0/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db/seg0/cc60.dat
> for container
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack(TxnHandler.java:2172)
> ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.setConf(TxnHandler.java:228)
> ~[hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.txn.TxnUtils.getTxnStore(TxnUtils.java:96)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getTxnHandler(HiveMetaStore.java:557)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.heartbeat(HiveMetaStore.java:5902)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_25]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_25]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_25]
> at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at com.sun.proxy.$Proxy111.heartbeat(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.heartbeat(HiveMetaStoreClient.java:2140)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_25]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_25]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_25]
> at java.lang.reflect.Method.invoke(Method.java:483) ~[?:1.8.0_25]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
> [hive-metastore-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at com.sun.proxy.$Proxy112.heartbeat(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$SynchronizedMetaStoreClient.heartbeat(DbTxnManager.java:663)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.heartbeat(DbTxnManager.java:423)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager$Heartbeater.run(DbTxnManager.java:633)
> [hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_25]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_25]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_25]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_25]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_25]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_25]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
> Caused by: java.sql.SQLException: Exception during creation of file
>
[jira] [Commented] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
[
https://issues.apache.org/jira/browse/HIVE-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331864#comment-15331864
]
Ashutosh Chauhan commented on HIVE-13822:
-
[~hsubramaniyan] Any update on this one?
> TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot
> parse COLUMN_STATS
> --
>
> Key: HIVE-13822
> URL: https://issues.apache.org/jira/browse/HIVE-13822
> Project: Hive
> Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13822.1.patch
>
>
> Thanks to [~jcamachorodriguez] for uncovering this issue as part of
> HIVE-13269. StatsSetupConst.areColumnStatsUptoDate() is used to check whether
> stats are up-to-date. In case of PerfCliDriver, ‘false’ (thus, not
> up-to-date) is returned and the following debug message in the logs:
> {code}
> In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
> StatsSetupConst)
> {code}
> Looks like the issue started happening after HIVE-12261 went in.
> The fix would be to replace
> {color:red}COLUMN_STATS_ACCURATE,true{color}
> with
> {color:green}COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}{color}
> where key, value are the column names.
> in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331824#comment-15331824 ] Ashutosh Chauhan commented on HIVE-14002: - make sense.. +1 pending tests > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.01.patch, HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[ https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331814#comment-15331814 ] Yongzhi Chen commented on HIVE-14015: - Tested in CDH kerberized cluster with sentry. [~ctang.ma], could you review the code? > SMB MapJoin failed for Hive on Spark when kerberized > > > Key: HIVE-14015 > URL: https://issues.apache.org/jira/browse/HIVE-14015 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.1.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14015.1.patch > > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can be issued only with kerberos or web authentication > It could be reproduced: > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.execution.engine=spark; > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[ https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14015: Status: Patch Available (was: Open) patch 1 fix the issue by put mapreduce.job.credentials.binary to JobConf Need code review. > SMB MapJoin failed for Hive on Spark when kerberized > > > Key: HIVE-14015 > URL: https://issues.apache.org/jira/browse/HIVE-14015 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 2.0.0, 1.1.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14015.1.patch > > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can be issued only with kerberos or web authentication > It could be reproduced: > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.execution.engine=spark; > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14015) SMB MapJoin failed for Hive on Spark when kerberized
[ https://issues.apache.org/jira/browse/HIVE-14015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-14015: Attachment: HIVE-14015.1.patch SMB mapredlocalwork need to set HADOOP_TOKEN_FILE_LOCATION to JobConf > SMB MapJoin failed for Hive on Spark when kerberized > > > Key: HIVE-14015 > URL: https://issues.apache.org/jira/browse/HIVE-14015 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 1.1.0, 2.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen > Attachments: HIVE-14015.1.patch > > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token > can be issued only with kerberos or web authentication > It could be reproduced: > 1) prepare sample data: > a=1 > while [[ $a -lt 100 ]]; do echo $a ; let a=$a+1; done > data > 2) prepare source hive table: > CREATE TABLE `s`(`c` string); > load data local inpath 'data' into table s; > 3) prepare the bucketed table: > set hive.enforce.bucketing=true; > set hive.enforce.sorting=true; > CREATE TABLE `t`(`c` string) CLUSTERED BY (c) SORTED BY (c) INTO 5 BUCKETS; > insert into t select * from s; > 4) reproduce this issue: > SET hive.execution.engine=spark; > SET hive.auto.convert.sortmerge.join = true; > SET hive.auto.convert.sortmerge.join.bigtable.selection.policy = > org.apache.hadoop.hive.ql.optimizer.LeftmostBigTableSelectorForAutoSMJ; > SET hive.auto.convert.sortmerge.join.noconditionaltask = true; > SET hive.optimize.bucketmapjoin = true; > SET hive.optimize.bucketmapjoin.sortedmerge = true; > select * from t join t t1 on t.c=t1.c; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14013) Describe table doesn't show unicode properly
[
https://issues.apache.org/jira/browse/HIVE-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331761#comment-15331761
]
Peter Vary commented on HIVE-14013:
---
Hi,
I have stumbled upon your patch, and I think it would be useful to apply your
escaping to every comment field, like:
+++
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
(revision )
@@ -231,7 +233,7 @@
appendColumnStatsNoFormatting(colBuffer, "", "", "", "", "", "", "",
"");
}
}
-colBuffer.append(comment == null ? "" : comment);
+colBuffer.append(comment == null ? "" :
HiveStringUtils.escapeJava(comment));
colBuffer.append(LINE_DELIM);
}
@@ -266,7 +268,7 @@
IndexType indexType = HiveIndex.getIndexTypeByClassName(indexHandlerClass);
indexColumns.add(indexType.getName());
-indexColumns.add(index.getParameters().get("comment"));
+
indexColumns.add(HiveStringUtils.escapeJava(index.getParameters().get("comment")));
formatOutput(indexColumns.toArray(new String[0]), indexInfo);
Regards,
Peter
> Describe table doesn't show unicode properly
>
>
> Key: HIVE-14013
> URL: https://issues.apache.org/jira/browse/HIVE-14013
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14013.1.patch
>
>
> Describe table output will show comments incorrectly rather than the unicode
> itself.
> {noformat}
> hive> desc formatted t1;
> # Detailed Table Information
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> comment \u8868\u4E2D\u6587\u6D4B\u8BD5
> numFiles0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14014) zero length file is being created for empty bucket in tez mode (II)
[
https://issues.apache.org/jira/browse/HIVE-14014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331553#comment-15331553
]
Hive QA commented on HIVE-14014:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12810695/HIVE-14014.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 10230 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/126/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/126/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-126/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12810695 - PreCommit-HIVE-MASTER-Build
> zero length file is being created for empty bucket in tez mode (II)
> ---
>
> Key: HIVE-14014
> URL: https://issues.apache.org/jira/browse/HIVE-14014
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14014.01.patch, HIVE-14014.02.patch
>
>
> The same problem happens when source table is not empty, e.g,, when "limit 0"
> is not there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331524#comment-15331524 ] Matt McCline commented on HIVE-14000: - [~sershe] Thank you for the RB comment. I fixed the problem and resubmitted a patch. > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Attachment: HIVE-14000.03.patch > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: Patch Available (was: In Progress) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14000) (ORC) Changing a numeric type column of a partitioned table to lower type set values to something other than 'NULL'
[ https://issues.apache.org/jira/browse/HIVE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14000: Status: In Progress (was: Patch Available) > (ORC) Changing a numeric type column of a partitioned table to lower type set > values to something other than 'NULL' > --- > > Key: HIVE-14000 > URL: https://issues.apache.org/jira/browse/HIVE-14000 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14000.01.patch, HIVE-14000.03.patch > > > When an integer column is changed to a type that is smaller (e.g. bigint to > int) and set hive.metastore.disallow.incompatible.col.type.changes=false, the > data is clipped instead of being NULL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331503#comment-15331503 ] Jesus Camacho Rodriguez edited comment on HIVE-14002 at 6/15/16 10:29 AM: -- Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I expect that the operators are properly classified and it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method checks that). was (Author: jcamachorodriguez): Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I would expect that the operators are properly classified and it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method checks that). > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.01.patch, HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331503#comment-15331503 ] Jesus Camacho Rodriguez edited comment on HIVE-14002 at 6/15/16 10:30 AM: -- Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator and the logic seems sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method checks that). was (Author: jcamachorodriguez): Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I expect that the operators are properly classified and it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method checks that). > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.01.patch, HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14002: --- Attachment: HIVE-14002.01.patch Reuploading same patch to trigger QA. > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.01.patch, HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331503#comment-15331503 ] Jesus Camacho Rodriguez edited comment on HIVE-14002 at 6/15/16 10:29 AM: -- Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I would expect that the operators are properly classified and it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method checks that). was (Author: jcamachorodriguez): Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I would expect that it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method does that). > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.01.patch, HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14002: --- Status: Open (was: Patch Available) > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14002 started by Jesus Camacho Rodriguez. -- > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14002: --- Status: Patch Available (was: In Progress) > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14002) Extend limit propagation to subsequent RS operators
[ https://issues.apache.org/jira/browse/HIVE-14002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331503#comment-15331503 ] Jesus Camacho Rodriguez commented on HIVE-14002: Thanks for reviewing [~ashutoshc]. In fact, _operator.acceptLimitPushdown()_ is used to bail out if the operator does not accept the limit pushdown. This is the same logic that was used for pushing it from the Limit operator, thus I would expect that it is sound. Further, if we cannot backtrack a column from the top RS to the bottom RS, we bail out too (in fact, the new _checkKeys_ method does that). > Extend limit propagation to subsequent RS operators > --- > > Key: HIVE-14002 > URL: https://issues.apache.org/jira/browse/HIVE-14002 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 2.2.0 >Reporter: Nita Dembla >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14002.patch > > > In some occasions, for instance when RS dedup does not kick in, it is useful > to propagate the limit to subsequent RS operators, as this will reduce > intermediary results and impact performance. This issue covers that extension. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14017) Compaction failed when run on ACID table with extended schema
[
https://issues.apache.org/jira/browse/HIVE-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Dai Thanh updated HIVE-14017:
--
Description:
Create an ACID table, insert some data into the table. Then we extend the
schema of the table by adding a column at the end, and add data to the table
with the extended schema.
{code:borderStyle=solid}
drop table if exists test purge;
create table test (
a int,
b int
)
clustered by (a) into 10 buckets
stored as orc
tblproperties ('transactional' = 'true');
insert into test values (1, 1), (2, 2), (3, 3);
insert into test values (4, 4), (5, 5), (6, 6);
alter table test add columns (c int);
insert into test values (10, 10, 10), (11, 11, 11), (12, 12, 12);
{code}
We then run compaction on the table:
{code}alter table test compact 'major';{code}
However, the compaction job fails with the following exception:
{code}
2016-06-15 09:54:52,517 INFO [IPC Server handler 5 on 25906]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1465960802609_0030_m_08_0 is : 0.0
2016-06-15 09:54:52,525 FATAL [IPC Server handler 4 on 25906]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1465960802609_0030_m_08_0 - exited : java.io.IOException: subtype 9
exceeds the included array size 9 fileTypes [kind: STRUCT
subtypes: 1
subtypes: 2
subtypes: 3
subtypes: 4
subtypes: 5
subtypes: 6
fieldNames: "operation"
fieldNames: "originalTransaction"
fieldNames: "bucket"
fieldNames: "rowId"
fieldNames: "currentTransaction"
fieldNames: "row"
, kind: INT
, kind: LONG
, kind: INT
, kind: LONG
, kind: LONG
, kind: STRUCT
subtypes: 7
subtypes: 8
subtypes: 9
fieldNames: "_col0"
fieldNames: "_col1"
fieldNames: "_col2"
, kind: INT
, kind: INT
, kind: INT
] schemaTypes [kind: STRUCT
subtypes: 1
subtypes: 2
subtypes: 3
subtypes: 4
subtypes: 5
subtypes: 6
fieldNames: "operation"
fieldNames: "originalTransaction"
fieldNames: "bucket"
fieldNames: "rowId"
fieldNames: "currentTransaction"
fieldNames: "row"
, kind: INT
, kind: LONG
, kind: INT
, kind: LONG
, kind: LONG
, kind: STRUCT
subtypes: 7
subtypes: 8
subtypes: 9
fieldNames: "_col0"
fieldNames: "_col1"
fieldNames: "_col2"
, kind: INT
, kind: INT
, kind: INT
] innerStructSubtype -1
at
org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2066)
at
org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
at
org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.(TreeReaderFactory.java:2072)
at
org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory.createTreeReader(TreeReaderFactory.java:2492)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:219)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:179)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:476)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRawReader(OrcInputFormat.java:1463)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:573)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorMap.map(CompactorMR.java:552)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{code}
was:
Create an ACID table, insert the data into the table, then extend the schema of
the table by adding a column at the end, then add data to the table with the
extended schema.
{code:borderStyle=solid}
drop table if exists test purge;
create table test (
a int,
b int
)
clustered by (a) into 10 buckets
stored as orc
tblproperties ('transactional' = 'true');
insert into test values (1, 1), (2, 2), (3, 3);
insert into test values (4, 4), (5, 5), (6, 6);
alter table test add columns (c int);
insert into test values (10, 10, 10), (11, 11, 11), (12, 12, 12);
{code}
We then run compaction on the table:
{code}alter table test compact 'major';{code}
However, the compaction job fails with the following exception:
{code}
2016-06-15 09:54:52,517 INFO [IPC Server handler 5 on 25906]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1465960802609_0030_m_08_0 is : 0.0
2016-06-15 09:54:52,525 FATAL [IPC Server handler 4
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331428#comment-15331428 ] Prasanth Jayachandran commented on HIVE-13985: -- One more change is that local cache is updated to hard reference cache for better predictability. > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-2.1.patch, > HIVE-13985.1.patch, HIVE-13985.2.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13956) LLAP: external client output is writing to channel before it is writable again
[ https://issues.apache.org/jira/browse/HIVE-13956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13956: -- Attachment: HIVE-13956.2.patch rebased. > LLAP: external client output is writing to channel before it is writable again > -- > > Key: HIVE-13956 > URL: https://issues.apache.org/jira/browse/HIVE-13956 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13956.1.patch, HIVE-13956.2.patch > > > Rows are being written/flushed on the output channel without checking if the > channel is writable. Introduce a writability check/wait. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331425#comment-15331425 ] Prasanth Jayachandran commented on HIVE-13985: -- Here is a summary of changes 1) Added a new proto object FileTail (This is from c++ version https://github.com/apache/orc/blob/master/proto/orc_proto.proto#L227) 2) OrcTail is the class that wraps FileTail and serialized footer (ByteBuffer). - FileTail is used in OrcSplit to serialize minimal footer in OrcSplit (strips off file level column statistics which is not required in task side) - OrcTail is cached by LocalCache (reconstructs every other objects from this) 3) Encodes file length in the OrcSplit which avoids 1 file system call to get file status on task side 4) Bunch of file system counters based unit tests to make sure we are not making excessive file system calls [~sershe] could you please review the changes? > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-2.1.patch, > HIVE-13985.1.patch, HIVE-13985.2.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13985: - Attachment: HIVE-13985-branch-1.patch > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-2.1.patch, > HIVE-13985.1.patch, HIVE-13985.2.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13985) ORC improvements for reducing the file system calls in task side
[ https://issues.apache.org/jira/browse/HIVE-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13985: - Attachment: (was: HIVE-13985-branch-1.patch) > ORC improvements for reducing the file system calls in task side > > > Key: HIVE-13985 > URL: https://issues.apache.org/jira/browse/HIVE-13985 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13985-branch-1.patch, HIVE-13985-branch-2.1.patch, > HIVE-13985.1.patch, HIVE-13985.2.patch > > > HIVE-13840 fixed some issues with addition file system invocations during > split generation. Similarly, this jira will fix issues with additional file > system invocations on the task side. To avoid reading footers on the task > side, users can set hive.orc.splits.include.file.footer to true which will > serialize the orc footers on the splits. But this has issues with serializing > unwanted information like column statistics and other metadata which are not > really required for reading orc split on the task side. We can reduce the > payload on the orc splits by serializing only the minimum required > information (stripe information, types, compression details). This will > decrease the payload on the orc splits and can potentially avoid OOMs in > application master (AM) during split generation. This jira also address other > issues concerning the AM cache. The local cache used by AM is soft reference > cache. This can introduce unpredictability across multiple runs of the same > query. We can cache the serialized footer in the local cache and also use > strong reference cache which should avoid memory pressure and will have > better predictability. > One other improvement that we can do is when > hive.orc.splits.include.file.footer is set to false, on the task side we make > one additional file system call to know the size of the file. If we can > serialize the file length in the orc split this can be avoided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13648) ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR, or DECIMAL when maxLength or precision/scale is different
[
https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331407#comment-15331407
]
Matt McCline edited comment on HIVE-13648 at 6/15/16 8:56 AM:
--
[~prasanth_j]
Hi Prasanth,
Thanks for you review.
When converting from CHAR(N) to CHAR(M), the
StringGroupFromStringGroupTreeReader class is used. It calls
convertStringGroupVectorElement to do any necessary trimming/length adjustment.
The Q files have cases for converting to smaller and larger maxLength, etc.
Yes, I saw that when the row reader stuff was removed a bunch of not used
members / parameters appeared. I had already removed them in the patch that
Sergey is currently reviewing.
Matt
was (Author: mmccline):
[~prasanth_j]
Hi Prasanth,
Thanks for you review.
When converting from CHAR(N) to CHAR(M), the
StringGroupFromStringGroupTreeReader class is used. It calls
convertStringGroupVectorElement to do any necessary trimming/length adjustment.
Yes, I saw that when the row reader stuff was removed a bunch of not used
members / parameters appeared. I had already removed them in the patch that
Sergey is currently reviewing.
Matt
> ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR,
> or DECIMAL when maxLength or precision/scale is different
> --
>
> Key: HIVE-13648
> URL: https://issues.apache.org/jira/browse/HIVE-13648
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13648.01.patch, HIVE-13648.02.patch
>
>
> E.g. when a data file is copied in has a VARCHAR maxLength that doesn't match
> the DDL's maxLength. This error is produced:
> {code}
> java.io.IOException: ORC does not support type conversion from file type
> varchar(145) (36) to reader type varchar(114) (36)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13648) ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR, or DECIMAL when maxLength or precision/scale is different
[
https://issues.apache.org/jira/browse/HIVE-13648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331409#comment-15331409
]
Prasanth Jayachandran commented on HIVE-13648:
--
[~mmccline] thanks for the clarification!
LGTM, +1
> ORC Schema Evolution doesn't support same type conversion for VARCHAR, CHAR,
> or DECIMAL when maxLength or precision/scale is different
> --
>
> Key: HIVE-13648
> URL: https://issues.apache.org/jira/browse/HIVE-13648
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13648.01.patch, HIVE-13648.02.patch
>
>
> E.g. when a data file is copied in has a VARCHAR maxLength that doesn't match
> the DDL's maxLength. This error is produced:
> {code}
> java.io.IOException: ORC does not support type conversion from file type
> varchar(145) (36) to reader type varchar(114) (36)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
