[jira] [Created] (HIVE-14670) org.apache.hadoop.hive.ql.TestMTQueries failure
Hari Sankar Sivarama Subramaniyan created HIVE-14670: Summary: org.apache.hadoop.hive.ql.TestMTQueries failure Key: HIVE-14670 URL: https://issues.apache.org/jira/browse/HIVE-14670 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Introduced by HIVE-14627. We used to have a common q_init file for MR and CliDriver tests till HIVE-14627 was committed. Now, that the init files are separate and join1.q and groupby2.q are run as part of minimr tests, we cannot use these tests to test multi-threaded queries with the same setup file because they would result in different stats (due to the way the init scripts are written). The easy fix would be to substitute join1.q and groupby2.q with 2 files that actually run in CliDriver mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14665) vector_join_part_col_char.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-14665: Summary: vector_join_part_col_char.q failure Key: HIVE-14665 URL: https://issues.apache.org/jira/browse/HIVE-14665 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Happens 100% of the time. Looks like a missed golden file update from HIVE-14502. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
Hari Sankar Sivarama Subramaniyan created HIVE-14612: Summary: org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure Key: HIVE-14612 URL: https://issues.apache.org/jira/browse/HIVE-14612 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Failing for some time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14578) Fixes to TestEncryptedHDFSCliDriver
Hari Sankar Sivarama Subramaniyan created HIVE-14578: Summary: Fixes to TestEncryptedHDFSCliDriver Key: HIVE-14578 URL: https://issues.apache.org/jira/browse/HIVE-14578 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan 900s runtime. One of the stragglers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14552) TestParseNegative fix
Hari Sankar Sivarama Subramaniyan created HIVE-14552: Summary: TestParseNegative fix Key: HIVE-14552 URL: https://issues.apache.org/jira/browse/HIVE-14552 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan 1300s runtime. Straggler towards the end of the build. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14546) TestMiniTezCliDriver plan diffs
Hari Sankar Sivarama Subramaniyan created HIVE-14546:
Summary: TestMiniTezCliDriver plan diffs
Key: HIVE-14546
URL: https://issues.apache.org/jira/browse/HIVE-14546
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
The following tests are failing consistently for the last some days in
precommit runs:
{code}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez_join_hash]
56 sec 5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_2]
35 sec 19
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1]
58 sec 19
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part1]
2.8 sec 19
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[transform_ppr1]
2.4 sec 19
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-14520) We should set a timeout for the blocking calls in TestMsgBusConnection
Hari Sankar Sivarama Subramaniyan created HIVE-14520: Summary: We should set a timeout for the blocking calls in TestMsgBusConnection Key: HIVE-14520 URL: https://issues.apache.org/jira/browse/HIVE-14520 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan consumer.receive() is a blocking call and if it fails, it will block for ever. Need to set timeout at the bare minimum to force the test to fail incase of failure rather than timing out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14507) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters failure
Hari Sankar Sivarama Subramaniyan created HIVE-14507: Summary: org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters failure Key: HIVE-14507 URL: https://issues.apache.org/jira/browse/HIVE-14507 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Fails locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14506) TestQueryLifeTimeHook hang
Hari Sankar Sivarama Subramaniyan created HIVE-14506: Summary: TestQueryLifeTimeHook hang Key: HIVE-14506 URL: https://issues.apache.org/jira/browse/HIVE-14506 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan The test hangs locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14505) Analyze org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching failure
Hari Sankar Sivarama Subramaniyan created HIVE-14505: Summary: Analyze org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching failure Key: HIVE-14505 URL: https://issues.apache.org/jira/browse/HIVE-14505 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Flaky test failure. Fails ~50% of the time locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14020) Hive MS restart failed during EU with ORA-00922 error as part of DB schema upgrade
Hari Sankar Sivarama Subramaniyan created HIVE-14020:
Summary: Hive MS restart failed during EU with ORA-00922 error as
part of DB schema upgrade
Key: HIVE-14020
URL: https://issues.apache.org/jira/browse/HIVE-14020
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
The underlying failure seems to be visible from --verbose :
{noformat}
Metastore connection URL:jdbc:oracle:thin:@//172.22.66.99:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 2.0.0 to 2.1.0
Upgrade script upgrade-2.0.0-to-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.66.99:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.4.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.66.99:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.66.99:1521/XE> SELECT 'Upgrading MetaStore schema
from 2.0.0 to 2.1.0' AS Status from dual
+-+--+
| STATUS |
+-+--+
| Upgrading MetaStore schema from 2.0.0 to 2.1.0 |
+-+--+
1 row selected (0.072 seconds)
0: jdbc:oracle:thin:@//172.22.66.99:1521/XE> CREATE TABLE IF NOT EXISTS
KEY_CONSTRAINTS ( CHILD_CD_ID NUMBER, CHILD_INTEGER_IDX NUMBER, CHILD_TBL_ID
NUMBER, PARENT_CD_ID NUMBER NOT NULL, PARENT_INTEGER_IDX ^M NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL, POSITION NUMBER NOT NULL, CONSTRAINT_NAME
VARCHAR(400) NOT NULL, CONSTRAINT_TYPE NUMBER NOT NULL, UPDATE_RULE NUMBER,
DELETE_RULE NUMBER, ENABLE_VALIDATE_REL ^MY NUMBER NOT NULL )
Error: ORA-00922: missing or invalid option (state=42000,code=922)
Closing: 0: jdbc:oracle:thin:@//172.22.66.99:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
Underlying cause: java.io.IOException : Schema script failed, errorcode 2
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:250)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:218)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:500)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:245)
... 8 more
*** schemaTool failed ***
{noformat}
At the face of it, it looks like bad ^Ms got added from the actual script (
034-HIVE-13076.oracle.sql ) that's provided:
{noformat}
CREATE TABLE IF NOT EXISTS KEY_CONSTRAINTS
(
CHILD_CD_ID NUMBER,
CHILD_INTEGER_IDX NUMBER,
CHILD_TBL_ID NUMBER,
PARENT_CD_ID NUMBER NOT NULL,
PARENT_INTEGER_IDX NUMBER NOT NULL,
PARENT_TBL_ID NUMBER NOT NULL,
POSITION NUMBER NOT NULL,
CONSTRAINT_NAME VARCHAR(400) NOT NULL,
CONSTRAINT_TYPE NUMBER NOT NULL,
UPDATE_RULE NUMBER,
DELETE_RULE NUMBER,
ENABLE_VALIDATE_RELY NUMBER NOT NULL
) ;
ALTER TABLE KEY_CONSTRAINTS ADD CONSTRAINT CONSTRAINTS_PK PRIMARY KEY
(CONSTRAINT_NAME, POSITION);
CREATE INDEX CONSTRAINTS_PT_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13911) load inpath fails throwing org.apache.hadoop.security.AccessControlException
Hari Sankar Sivarama Subramaniyan created HIVE-13911: Summary: load inpath fails throwing org.apache.hadoop.security.AccessControlException Key: HIVE-13911 URL: https://issues.apache.org/jira/browse/HIVE-13911 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-13911.1.patch Similar to HIVE-13857 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13857) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II
Hari Sankar Sivarama Subramaniyan created HIVE-13857: Summary: insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException - II Key: HIVE-13857 URL: https://issues.apache.org/jira/browse/HIVE-13857 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HIVE-13810 missed a fix, tracking it here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13822) TestPerfCliDriver throws warning in StatsSetupConst that JsonParser cannot parse COLUMN_STATS
Hari Sankar Sivarama Subramaniyan created HIVE-13822:
Summary: TestPerfCliDriver throws warning in StatsSetupConst that
JsonParser cannot parse COLUMN_STATS
Key: HIVE-13822
URL: https://issues.apache.org/jira/browse/HIVE-13822
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Thanks to [~jcamachorodriguez] for uncovering this issue as part of HIVE-13269.
StatsSetupConst.areColumnStatsUptoDate() is used to check whether stats are
up-to-date. In case of PerfCliDriver, ‘false’ (thus, not up-to-date) is
returned and the following debug message in the logs:
{code}
In StatsSetupConst, JsonParser can not parse COLUMN_STATS. (line 190 in
StatsSetupConst)
{code}
Looks like the issue started happening after HIVE-12261 went in.
The fix would be to replace
{{COLUMN_STATS_ACCURATE,true}}
with
{{COLUMN_STATS_ACCURATE,{"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}}}
in data/files/tpcds-perf/metastore_export/csv/TABLE_PARAMS.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
Hari Sankar Sivarama Subramaniyan created HIVE-13810:
Summary: insert overwrite select from some table fails throwing
org.apache.hadoop.security.AccessControlException
Key: HIVE-13810
URL: https://issues.apache.org/jira/browse/HIVE-13810
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{noformat} CREATE EXTERNAL TABLE Batters_txt(
Player STRING ,
Team STRING ,
League STRING ,
Year SMALLINT,
Games DOUBLE,
AB DOUBLE,
R DOUBLE,
H DOUBLE,
Doubles DOUBLE,
Triples DOUBLE,
HR DOUBLE,
RBI DOUBLE,
SB DOUBLE,
CS DOUBLE,
BB DOUBLE,
SO DOUBLE,
IBB DOUBLE,
HBP DOUBLE,
SH DOUBLE,
SF DOUBLE,
GIDP DOUBLE
)
location '/user/tableau/Batters';
drop table if exists Batters;
CREATE TABLE Batters (
Player STRING ,
Team STRING ,
League STRING ,
Year SMALLINT,
Games DOUBLE,
AB DOUBLE,
R DOUBLE,
H DOUBLE,
Doubles DOUBLE,
Triples DOUBLE,
HR DOUBLE,
RBI DOUBLE,
SB DOUBLE,
CS DOUBLE,
BB DOUBLE,
SO DOUBLE,
IBB DOUBLE,
HBP DOUBLE,
SH DOUBLE,
SF DOUBLE,
GIDP DOUBLE
)
STORED AS orc tblproperties ("orc.compress"="SNAPPY");
insert overwrite table Batters select * from Batters_txt;
{noformat}
runs into the following error:
{code}
2016-05-18T19:59:00,883 ERROR [HiveServer2-Background-Pool: Thread-306]:
operation.Operation (:()) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask.
org.apache.hadoop.security.AccessControlException: User does not belong to hdfs
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:644)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
at
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:290)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:303)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.security.AccessControlException: User does not belong to hdfs
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:88)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1706)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:818)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:472)
at
[jira] [Created] (HIVE-13784) Hive Metastore start failed on Oracle DB
Hari Sankar Sivarama Subramaniyan created HIVE-13784:
Summary: Hive Metastore start failed on Oracle DB
Key: HIVE-13784
URL: https://issues.apache.org/jira/browse/HIVE-13784
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
During hive metastore start, hive is trying to execute this script
{{hive-schema-2.1.0.oracle.sql}}:
{code}
Metastore connection URL:jdbc:oracle:thin:@//172.22.114.220:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User: hiveuser
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.2.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE (
SEQ
UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
No rows affected (0.208 seconds)
..
{code}
but inthe end of this script we are getting Error:
{code}
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX
CONSTRAINTS_PARENT_T
BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
Error: ORA-00972: identifier is too long (state=42000,code=972)
Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
FAILED! Metastore state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
... 8 more
*** schemaTool failed ***
{code}
We have a problem with too long identifier: {{CONSTRAINTS_PARENT_TBL_ID_INDEX}}
(31 symbol), it should be 30 symbols of max length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13715) Show create table should show the primary keys/foreign keys associated with the table
Hari Sankar Sivarama Subramaniyan created HIVE-13715: Summary: Show create table should show the primary keys/foreign keys associated with the table Key: HIVE-13715 URL: https://issues.apache.org/jira/browse/HIVE-13715 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Follow up for HIVE-13598 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13608) We should provide better error message while constraints with duplicate names are created
Hari Sankar Sivarama Subramaniyan created HIVE-13608:
Summary: We should provide better error message while constraints
with duplicate names are created
Key: HIVE-13608
URL: https://issues.apache.org/jira/browse/HIVE-13608
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code}
PREHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
novalidate)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@t1
POSTHOOK: query: create table t1(x int, constraint pk1 primary key (x) disable
novalidate)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@t1
PREHOOK: query: create table t2(x int, constraint pk1 primary key (x) disable
novalidate)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@t2
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct
MetaStore DB connections, we don't support retries at the client level.)
{code}
In the above case, it seems like useful error message is lost. It looks like a
generic problem with metastore server/client exception handling and message
propagation. Seems like exception parsing logic of
RetryingMetaStoreClient::invoke() needs to be updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13601) Deadline is not registered at some places in HiveMetastore before the ObjectStore/MetaStoreDirectSQL APIs are invoked
Hari Sankar Sivarama Subramaniyan created HIVE-13601:
Summary: Deadline is not registered at some places in
HiveMetastore before the ObjectStore/MetaStoreDirectSQL APIs are invoked
Key: HIVE-13601
URL: https://issues.apache.org/jira/browse/HIVE-13601
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
HIVE-13542 exposed this issue as shown below when making Metastore API calls
via CBO code path.
{code}
16-04-21T11:45:55,201 DEBUG [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:timingTrace(864)) -
Direct SQL query in 14.825303ms + 0.086336ms, the query is [select
"COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE",
"DOUBLE_LOW_VALU\
E", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE",
"NUM_NULLS", "NUM_DISTINCTS", "AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES",
"NUM_FALSES", "LAST_ANALYZED" from "TAB_COL_STATS" where "DB_NAME" = ? and
"TABLE_NAME" = ? and "COLUMN_NAME" in (...)]
2016-04-21T11:45:55,242 WARN [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]:
metastore.ObjectStore (ObjectStore.java:handleDirectSqlError(2602)) - Direct
SQL failed, falling back to ORM
org.apache.hadoop.hive.metastore.api.MetaException: The threadlocal Deadline is
null, please register it first.
at
org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:178)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:1717)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1201)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6849)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6846)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2567)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6845)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6839)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4361)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1613)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:758)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:302)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:417)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:410)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:158)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.7.0_79]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[?:1.7.0_79]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[jira] [Created] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
Hari Sankar Sivarama Subramaniyan created HIVE-13598: Summary: Describe extended table should show the primary keys/foreign keys associated with the table Key: HIVE-13598 URL: https://issues.apache.org/jira/browse/HIVE-13598 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan After HIVE-13290 is committed, we need to show the constraints as part of table description when extended label is used. Currently, the constraints would not be shown as part of table description since Constraint is a separate entity. The purpose of the jira is to modify Hive.describeTable() to enable the user to view the constraints associated with the table when the user does a "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
Hari Sankar Sivarama Subramaniyan created HIVE-13542: Summary: Missing stats for tables in TPCDS performance regression suite Key: HIVE-13542 URL: https://issues.apache.org/jira/browse/HIVE-13542 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan These are the tables whose stats are missing in data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt: * catalog_returns * catalog_sales * inventory * store_returns * store_sales * web_returns * web_sales Thanks to [~jcamachorodriguez] for discovering this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13407) Add more subtlety to TezCompiler Perf Logging
Hari Sankar Sivarama Subramaniyan created HIVE-13407: Summary: Add more subtlety to TezCompiler Perf Logging Key: HIVE-13407 URL: https://issues.apache.org/jira/browse/HIVE-13407 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We can add more subtlety to perf logging information in TezCompiler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
Hari Sankar Sivarama Subramaniyan created HIVE-13351: Summary: Support drop Primary Key/Foreign Key constraints Key: HIVE-13351 URL: https://issues.apache.org/jira/browse/HIVE-13351 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13350) Support Alter commands for Rely/NoRely novalidate
Hari Sankar Sivarama Subramaniyan created HIVE-13350: Summary: Support Alter commands for Rely/NoRely novalidate Key: HIVE-13350 URL: https://issues.apache.org/jira/browse/HIVE-13350 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13349) Metastore Changes : HS2 changes : API calls for retrieving primary keys and foreign keys information
Hari Sankar Sivarama Subramaniyan created HIVE-13349: Summary: Metastore Changes : HS2 changes : API calls for retrieving primary keys and foreign keys information Key: HIVE-13349 URL: https://issues.apache.org/jira/browse/HIVE-13349 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
Hari Sankar Sivarama Subramaniyan created HIVE-13290: Summary: Support primary keys/foreign keys constraint as part of create table command in Hive Key: HIVE-13290 URL: https://issues.apache.org/jira/browse/HIVE-13290 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13192) OOM error when MapJoinBytesTableContainer memory threshold is too low
Hari Sankar Sivarama Subramaniyan created HIVE-13192:
Summary: OOM error when MapJoinBytesTableContainer memory
threshold is too low
Key: HIVE-13192
URL: https://issues.apache.org/jira/browse/HIVE-13192
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
HIVE-11449 covered the scenario with HybridHashTableContainer, but we can still
the error as shown below:
{code}
Vertex failed, vertexName=Map 1, vertexId=vertex_1454464706407_0225_1_19,
diagnostics=[Task failed, taskId=task_1454464706407_0225_1_19_00,
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running
task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:333)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:177)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hive.serde2.WriteBuffers.nextBufferToWrite(WriteBuffers.java:206)
at org.apache.hadoop.hive.serde2.WriteBuffers.write(WriteBuffers.java:182)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryUtils.writeVLong(LazyBinaryUtils.java:411)
at org.apache.hadoop.hive.serde2.WriteBuffers.writeVLong(WriteBuffers.java:294)
at
org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.addRecordToList(BytesBytesMultiHashMap.java:540)
at
org.apache.hadoop.hive.ql.exec.persistence.BytesBytesMultiHashMap.put(BytesBytesMultiHashMap.java:219)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer.putRow(MapJoinBytesTableContainer.java:285)
at
org.apache.hadoop.hive.ql.exec.tez.HashTableLoader.load(HashTableLoader.java:114)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:190)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:216)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1051)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1055)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:37)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:316)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:163)
... 13 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13130) API calls for retrieving primary keys and foreign keys information
Hari Sankar Sivarama Subramaniyan created HIVE-13130: Summary: API calls for retrieving primary keys and foreign keys information Key: HIVE-13130 URL: https://issues.apache.org/jira/browse/HIVE-13130 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes getPrimaryKeys and getCrossReference API calls. We need to provide these interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13005) CBO: Calcite Operator To Hive Operator (Calcite Return Path): RexNode convert(ExprNodeConstantDesc literal) decimal support bug
Hari Sankar Sivarama Subramaniyan created HIVE-13005:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): RexNode convert(ExprNodeConstantDesc literal) decimal support bug
Key: HIVE-13005
URL: https://issues.apache.org/jira/browse/HIVE-13005
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
HIVE-8064 seems to have introduced this code in RexNodeConverter::convert(),
but the parameters look like they are wrongly passed :
{code}
RelDataType relType =
cluster.getTypeFactory().createSqlType(SqlTypeName.DECIMAL,
bd.scale(), unscaled.toString().length());
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12923) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_grouping_sets4.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12923:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): TestCliDriver groupby_grouping_sets4.q failure
Key: HIVE-12923
URL: https://issues.apache.org/jira/browse/HIVE-12923
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code}
EXPLAIN
SELECT * FROM
(SELECT a, b, count(*) from T1 where a < 3 group by a, b with cube) subq1
join
(SELECT a, b, count(*) from T1 where a < 3 group by a, b with cube) subq2
on subq1.a = subq2.a
{code}
Stack trace:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory.pruneJoinOperator(ColumnPrunerProcFactory.java:1110)
at
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory.access$400(ColumnPrunerProcFactory.java:85)
at
org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerJoinProc.process(ColumnPrunerProcFactory.java:941)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at
org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:237)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10176)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:472)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1168)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1256)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1094)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1082)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1129)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1103)
at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:10444)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets4(TestCliDriver.java:3313)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12924) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_ppr_multi_distinct.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12924:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): TestCliDriver groupby_ppr_multi_distinct.q failure
Key: HIVE-12924
URL: https://issues.apache.org/jira/browse/HIVE-12924
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code}
EXPLAIN EXTENDED
FROM srcpart src
INSERT OVERWRITE TABLE dest1
SELECT substr(src.key,1,1), count(DISTINCT substr(src.value,5)),
concat(substr(src.key,1,1),sum(substr(src.value,5))), sum(DISTINCT
substr(src.value, 5)), count(DISTINCT src.value)
WHERE src.ds = '2008-04-08'
GROUP BY substr(src.key,1,1)
{code}
Stack trace:
{code}
2016-01-25T14:27:56,694 DEBUG [4e6a139e-a78c-4f61-bb10-57af2b0d4381 main[]]:
parse.TypeCheckCtx (TypeCheckCtx.java:setError(159)) - Setting error: [Line
6:79 Expression not in GROUP BY key 'key'] from (tok_table_or_col src)
java.lang.Exception
at
org.apache.hadoop.hive.ql.parse.TypeCheckCtx.setError(TypeCheckCtx.java:159)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$ColumnExprProcessor.process(TypeCheckProcFactory.java:628)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:158)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:213)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:157)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10512)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10468)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genSelectLogicalPlan(CalcitePlanner.java:2920)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:3053)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:874)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:832)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:112)
[calcite-core-1.5.0.jar:1.5.0]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:971)
[calcite-core-1.5.0.jar:1.5.0]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:148)
[calcite-core-1.5.0.jar:1.5.0]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105)
[calcite-core-1.5.0.jar:1.5.0]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedHiveOPDag(CalcitePlanner.java:677)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:264)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10100)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:229)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
[jira] [Created] (HIVE-12866) Allow ReduceSinkDeDuplication to kick in when there are constant keys
Hari Sankar Sivarama Subramaniyan created HIVE-12866:
Summary: Allow ReduceSinkDeDuplication to kick in when there are
constant keys
Key: HIVE-12866
URL: https://issues.apache.org/jira/browse/HIVE-12866
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Currently in ReduceSinkDeDuplication.checkExprs() we have the following check
which prevents the optimization from kicking in as soon as we hit constant key.
We should lift this restriction as much as possible.
{code}
private Integer checkExprs(List ckeys, List
pkeys,
ReduceSinkOperator cRS, ReduceSinkOperator pRS) throws
SemanticException {
// If ckeys or pkeys have constant node expressions avoid the merge.
for (ExprNodeDesc ck : ckeys) {
if (ck instanceof ExprNodeConstantDesc) {
return null;
}
}
for (ExprNodeDesc pk : pkeys) {
if (pk instanceof ExprNodeConstantDesc) {
return null;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12802) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver.vector_join_filters.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12802:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): MiniTezCliDriver.vector_join_filters.q failure
Key: HIVE-12802
URL: https://issues.apache.org/jira/browse/HIVE-12802
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Discovered as part of running :
mvn test -Dtest=TestMiniTezCliDriver -Dqfile_regex=vector.*
-Dhive.cbo.returnpath.hiveop=true -Dtest.output.overwrite=true
{code}
2016-01-07T11:16:06,198 ERROR [657fd759-7643-467b-9bd0-17cb4958cb69 main[]]:
parse.CalcitePlanner (CalcitePlanner.java:genOPTree(309)) - CBO failed,
skipping CBO.
java.lang.IndexOutOfBoundsException: index (10) must be less than size (6)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
~[guava-14.0.1.jar:?]
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
~[guava-14.0.1.jar:?]
at
com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
~[guava-14.0.1.jar:?]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitInputRef(ExprNodeConverter.java:109)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitInputRef(ExprNodeConverter.java:79)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:112)
~[calcite-core-1.5.0.jar:1.5.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:128)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter.visitCall(ExprNodeConverter.java:79)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.calcite.rex.RexCall.accept(RexCall.java:107)
~[calcite-core-1.5.0.jar:1.5.0]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.convertToExprNode(HiveOpConverter.java:1153)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.translateJoin(HiveOpConverter.java:381)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.visit(HiveOpConverter.java:313)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.dispatch(HiveOpConverter.java:164)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.visit(HiveOpConverter.java:268)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.dispatch(HiveOpConverter.java:162)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.visit(HiveOpConverter.java:397)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.dispatch(HiveOpConverter.java:181)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.convert(HiveOpConverter.java:154)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedHiveOPDag(CalcitePlanner.java:688)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:266)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10094)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:231)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471)
[hive-exec-2.1.0-SNAPSHOT.jar:?]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12803) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver count.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12803:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): MiniTezCliDriver count.q failure
Key: HIVE-12803
URL: https://issues.apache.org/jira/browse/HIVE-12803
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code}
select a, count(distinct b), count(distinct c), sum(d) from abcd group by a;
{code}
Set hive.cbo.returnpath.hiveop=true;
{code}
java.lang.IndexOutOfBoundsException: Index: 5, Size: 5
at java.util.ArrayList.rangeCheck(ArrayList.java:635) ~[?:1.7.0_79]
at java.util.ArrayList.get(ArrayList.java:411) ~[?:1.7.0_79]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.genReduceSideGB1NoMapGB(HiveGBOpConvUtil.java:1060)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.genNoMapSideGBNoSkew(HiveGBOpConvUtil.java:473)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveGBOpConvUtil.translateGB(HiveGBOpConvUtil.java:304)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.visit(HiveOpConverter.java:398)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.dispatch(HiveOpConverter.java:181)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.optimizer.calcite.translator.HiveOpConverter.convert(HiveOpConverter.java:154)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedHiveOPDag(CalcitePlanner.java:688)
~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:266)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10094)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:231)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471)
[hive-exec-2.1.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
[hive-exec-2.1.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149)
[hive-exec-2.1.0-SNAPSHOT.jar:?]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
[hive-exec-2.1.0-SNAPSHOT.jar:?]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12805) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver skewjoin.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12805:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): MiniTezCliDriver skewjoin.q failure
Key: HIVE-12805
URL: https://issues.apache.org/jira/browse/HIVE-12805
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Set hive.cbo.returnpath.hiveop=true
{code}
FROM T1 a FULL OUTER JOIN T2 c ON c.key+1=a.key SELECT /*+ STREAMTABLE(a) */
sum(hash(a.key)), sum(hash(a.val)), sum(hash(c.key))
{code}
The stack trace:
{code}
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at
org.apache.hadoop.hive.ql.ppd.SyntheticJoinPredicate$JoinSynthetic.process(SyntheticJoinPredicate.java:183)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at
org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:43)
at
org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
at
org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
at
org.apache.hadoop.hive.ql.lib.PreOrderOnceWalker.walk(PreOrderOnceWalker.java:54)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.ppd.SyntheticJoinPredicate.transform(SyntheticJoinPredicate.java:100)
at
org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:236)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10170)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:231)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:471)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12806) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
Hari Sankar Sivarama Subramaniyan created HIVE-12806:
Summary: CBO: Calcite Operator To Hive Operator (Calcite Return
Path): MiniTezCliDriver vector_auto_smb_mapjoin_14.q failure
Key: HIVE-12806
URL: https://issues.apache.org/jira/browse/HIVE-12806
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Step to reproduce:
mvn test -Dtest=TestMiniTezCliDriver -Dqfile=vector_auto_smb_mapjoin_14.q
-Dhive.cbo.returnpath.hiveop=true -Dtest.output.overwrite=true
Query :
{code}
select count(*) from (
select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b
on a.key = b.key
) subq1
{code}
Stack trace :
{code}
2016-01-07T14:08:04,803 ERROR [da534038-d792-4d16-86e9-87b9f971adda main[]]:
SessionState (SessionState.java:printError(1010)) - Vertex failed,
vertexName=Map 1, vertexId=vertex_1452204324051_0001_33_00, diagnostics=[Vertex
vertex_1452204324051_0001_33_00 [Map 1] k\
illed/failed due to:AM_USERCODE_FAILURE, Exception in VertexManager,
vertex:vertex_1452204324051_0001_33_00 [Map 1], java.lang.RuntimeException:
java.lang.RuntimeException: Failed to load plan: null:
java.lang.IllegalArgumentException: java.net.URISyntaxException: \
Relative path in absolute URI: subq1:amerge.xml
at
org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:314)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventRootInputInitialized.invoke(VertexManager.java:624)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:645)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:640)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:640)
at
org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:629)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to load plan: null:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path
in absolute URI: subq1:amerge.xml
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMergeWork(Utilities.java:339)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.populateMapWork(SplitGrouper.java:260)
at
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:172)
at
org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex.onRootVertexInitialized(CustomPartitionVertex.java:277)
... 12 more
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: subq1:amerge.xml
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.(Path.java:172)
at org.apache.hadoop.fs.Path.(Path.java:94)
at
org.apache.hadoop.hive.ql.exec.Utilities.getPlanPath(Utilities.java:588)
at
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:387)
... 16 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
subq1:amerge.xml
at java.net.URI.checkPath(URI.java:1804)
at java.net.URI.(URI.java:752)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 20 more
]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12798) MiniTezCliDriver failures
Hari Sankar Sivarama Subramaniyan created HIVE-12798:
Summary: MiniTezCliDriver failures
Key: HIVE-12798
URL: https://issues.apache.org/jira/browse/HIVE-12798
Project: Hive
Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
As of 01/04/2016, the following tests fail in the MiniTezCliDriver mode when
the cbo return path is enabled. We need to fix them :
{code}
vector_leftsemi_mapjoin
vector_join_filters
vector_interval_mapjoin
vector_left_outer_join
vectorized_mapjoin
vector_inner_join
vectorized_context
tez_vector_dynpart_hashjoin_1
count
auto_sortmerge_join_6
skewjoin
vector_auto_smb_mapjoin_14
auto_join_filters
vector_outer_join0
vector_outer_join1
vector_outer_join2
vector_outer_join3
vector_outer_join4
vector_outer_join5
hybridgrace_hashjoin_1
vector_mapjoin_reduce
vectorized_nested_mapjoin
vector_left_outer_join2
vector_char_mapjoin1
vector_decimal_mapjoin
vectorized_dynamic_partition_pruning
vector_varchar_mapjoin1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12681) Improvements on HIVE-11107 to remove template for PerfCliDriver and update stats data file.
Hari Sankar Sivarama Subramaniyan created HIVE-12681: Summary: Improvements on HIVE-11107 to remove template for PerfCliDriver and update stats data file. Key: HIVE-12681 URL: https://issues.apache.org/jira/browse/HIVE-12681 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12675) PerfLogger should log performance metrics at debug level
Hari Sankar Sivarama Subramaniyan created HIVE-12675: Summary: PerfLogger should log performance metrics at debug level Key: HIVE-12675 URL: https://issues.apache.org/jira/browse/HIVE-12675 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan As more and more subcomponents are Hive (Tez, Optimizer) etc are using PerfLogger to track the performance metrics, it will be more meaningful to set the PerfLogger logging level to DEBUG. Otherwise, we will print the performance metrics unnecessarily for each and every query if the underlying subcomponent does not control the PerfLogging via a parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12666) PCRExprProcFactory.GenericFuncExprProcessor.process() aggressively removes dynamic partition pruner generated synthetic join predicates.
Hari Sankar Sivarama Subramaniyan created HIVE-12666:
Summary: PCRExprProcFactory.GenericFuncExprProcessor.process()
aggressively removes dynamic partition pruner generated synthetic join
predicates.
Key: HIVE-12666
URL: https://issues.apache.org/jira/browse/HIVE-12666
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Introduced by HIVE-11634. The original idea in HIVE-11634 was to remove the IN
partition conditions from the predicate list since the static dynamic
partitioning would kick in and push these predicates down to metastore.
However, the check is too aggressive and removes events such as below :
{code}
-Select Operator
- expressions: UDFToDouble(UDFToInteger((hr / 2))) (type:
double)
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 7 Basic stats:
COMPLETE Column stats: NONE
- Group By Operator
-keys: _col0 (type: double)
-mode: hash
-outputColumnNames: _col0
-Statistics: Num rows: 1 Data size: 7 Basic stats:
COMPLETE Column stats: NONE
-Dynamic Partitioning Event Operator
- Target Input: srcpart
- Partition key expr: UDFToDouble(hr)
- Statistics: Num rows: 1 Data size: 7 Basic stats:
COMPLETE Column stats: NONE
- Target column: hr
- Target Vertex: Map 1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12645) ConstantPropagateProcCtx.resolve() should use internal names instead of alias to match 2 columns from different row schemas
Hari Sankar Sivarama Subramaniyan created HIVE-12645: Summary: ConstantPropagateProcCtx.resolve() should use internal names instead of alias to match 2 columns from different row schemas Key: HIVE-12645 URL: https://issues.apache.org/jira/browse/HIVE-12645 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Currently, it seems that we look to match the ColumnInfo between the parent and the child rowschemas by calling rci = rs.getColumnInfo(tblAlias, alias) which might be a bit aggressive. i.e. we will lose opportunity to constant propogate even if the columns are the same but the alias in the rowschemas do not match. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12647) hive.mapred.mode=strict throws an error even if the final plan does not have cartesian product in it.
Hari Sankar Sivarama Subramaniyan created HIVE-12647:
Summary: hive.mapred.mode=strict throws an error even if the final
plan does not have cartesian product in it.
Key: HIVE-12647
URL: https://issues.apache.org/jira/browse/HIVE-12647
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
{code}
Vertex dependency in root stage
Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 11 (SIMPLE_EDGE)
Reducer 3 <- Map 12 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Map 13 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Map 14 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Map 15 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Map 16 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
Reducer 8 <- Map 17 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:100
Stage-1
Reducer 10
File Output Operator [FS_63]
compressed:false
Statistics:Num rows: 100 Data size: 143600 Basic stats: COMPLETE
Column stats: NONE
table:{"input
format:":"org.apache.hadoop.mapred.TextInputFormat","output
format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Limit [LIM_62]
Number of rows:100
Statistics:Num rows: 100 Data size: 143600 Basic stats: COMPLETE
Column stats: NONE
Select Operator [SEL_61]
|
outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14"]
| Statistics:Num rows: 127050 Data size: 182479129 Basic stats:
COMPLETE Column stats: NONE
|<-Reducer 9 [SIMPLE_EDGE]
Reduce Output Operator [RS_60]
key expressions:_col0 (type: string), _col1 (type:
string), _col2 (type: string)
sort order:+++
Statistics:Num rows: 127050 Data size: 182479129 Basic
stats: COMPLETE Column stats: NONE
value expressions:_col3 (type: bigint), _col4 (type:
double), _col5 (type: double), _col6 (type: double), _col7 (type: bigint),
_col8 (type: double), _col9 (type: double), _col10 (type: double), _col11
(type: bigint), _col12 (type: double), _col13 (type: double)
Select Operator [SEL_58]
outputColumnNames:["_col0","_col1","_col10","_col11","_col12","_col13","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9"]
Statistics:Num rows: 127050 Data size: 182479129 Basic
stats: COMPLETE Column stats: NONE
Group By Operator [GBY_57]
|
aggregations:["count(VALUE._col0)","avg(VALUE._col1)","stddev_samp(VALUE._col2)","count(VALUE._col3)","avg(VALUE._col4)","stddev_samp(VALUE._col5)","count(VALUE._col6)","avg(VALUE._col7)","stddev_samp(VALUE._col8)"]
| keys:KEY._col0 (type: string), KEY._col1 (type:
string), KEY._col2 (type: string)
|
outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11"]
| Statistics:Num rows: 127050 Data size: 182479129
Basic stats: COMPLETE Column stats: NONE
|<-Reducer 8 [SIMPLE_EDGE]
Reduce Output Operator [RS_56]
key expressions:_col0 (type: string), _col1
(type: string), _col2 (type: string)
Map-reduce partition columns:_col0 (type:
string), _col1 (type: string), _col2 (type: string)
sort order:+++
Statistics:Num rows: 254100 Data size: 364958258
Basic stats: COMPLETE Column stats: NONE
value expressions:_col3 (type: bigint), _col4
(type: struct), _col5 (type:
struct), _col6 (type: bigint), _col7
(type: struct), _col8 (type:
struct), _col9 (type: bigint), _col10
(type: struct), _col11 (type:
struct)
Group By Operator [GBY_55]
aggregations:["count(_col5)","avg(_col5)","stddev_samp(_col5)","count(_col10)","avg(_col10)","stddev_samp(_col10)","count(_col14)","avg(_col14)","stddev_samp(_col14)"]
keys:_col22 (type: string), _col24 (type:
string), _col25 (type: string)
[jira] [Created] (HIVE-12640) Allow StatsOptimizer to optimize the query for Constant GroupBy keys
Hari Sankar Sivarama Subramaniyan created HIVE-12640:
Summary: Allow StatsOptimizer to optimize the query for Constant
GroupBy keys
Key: HIVE-12640
URL: https://issues.apache.org/jira/browse/HIVE-12640
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code}
hive> select count('1') from src group by '1';
{code}
In the above query, while performing StatsOptimizer optimization we can safely
ignore the group by on the constant key '1' since the above query will return
the same result as "select count('1') from src".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12587) Support to add partitioned data set to TestPerfCliDriver
Hari Sankar Sivarama Subramaniyan created HIVE-12587: Summary: Support to add partitioned data set to TestPerfCliDriver Key: HIVE-12587 URL: https://issues.apache.org/jira/browse/HIVE-12587 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12586) Support for Performance regression test suite
Hari Sankar Sivarama Subramaniyan created HIVE-12586: Summary: Support for Performance regression test suite Key: HIVE-12586 URL: https://issues.apache.org/jira/browse/HIVE-12586 Project: Hive Issue Type: Task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Umbrella jira to add TestPerfCliDriver to the existing test framework -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12588) Support to use run TestPerfCliDriver test suite using HBase metastore db
Hari Sankar Sivarama Subramaniyan created HIVE-12588: Summary: Support to use run TestPerfCliDriver test suite using HBase metastore db Key: HIVE-12588 URL: https://issues.apache.org/jira/browse/HIVE-12588 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12570) Incorrect error message Expression not in GROUP BY key thrown instead of Invalid function
Hari Sankar Sivarama Subramaniyan created HIVE-12570:
Summary: Incorrect error message Expression not in GROUP BY key
thrown instead of Invalid function
Key: HIVE-12570
URL: https://issues.apache.org/jira/browse/HIVE-12570
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{source}
explain create table avg_salary_by_supervisor3 as select average(key) as
key_avg from src group by value;
{source}
We get the following stack trace :
{code}
FAILED: SemanticException [Error 10025]: Line 1:57 Expression not in GROUP BY
key 'key'
ERROR ql.Driver: FAILED: SemanticException [Error 10025]: Line 1:57 Expression
not in GROUP BY key 'key'
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:57 Expression not in
GROUP BY key 'key'
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10484)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10432)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3824)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3603)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8862)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8817)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9668)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9561)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10053)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:345)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10064)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:222)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:237)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:462)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:317)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1227)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1276)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1152)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:778)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:717)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}
Instead of the above error message, it be more appropriate to throw the below
error :
ERROR ql.Driver: FAILED: SemanticException [Error 10011]: Line 1:58 Invalid
function 'average'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12580) CalcitePlanner.genOpTree tries to run with cbo disable even after a semantic analysis exception
Hari Sankar Sivarama Subramaniyan created HIVE-12580:
Summary: CalcitePlanner.genOpTree tries to run with cbo disable
even after a semantic analysis exception
Key: HIVE-12580
URL: https://issues.apache.org/jira/browse/HIVE-12580
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
The following code in CalcitePlanner.genOPTree looks buggy :
{code}
if (!conf.getBoolVar(ConfVars.HIVE_IN_TEST) || isMissingStats
|| e instanceof CalciteSemanticException) {
reAnalyzeAST = true;
} else if (e instanceof SemanticException) {
throw (SemanticException) e;
} else if (e instanceof RuntimeException) {
throw (RuntimeException) e;
} else {
throw new SemanticException(e);
}
{code}
If we have missing stats and a Semantic exception thrown from from the try
block, we still go ahead and continue to generate a plan with cbo turned off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12526) PerfLogger for hive compiler and optimizer
Hari Sankar Sivarama Subramaniyan created HIVE-12526: Summary: PerfLogger for hive compiler and optimizer Key: HIVE-12526 URL: https://issues.apache.org/jira/browse/HIVE-12526 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan This jira is intended to use the perflogger to track compilation times and optimization times (calcite, tez compiler, physical compiler) etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12340) ExecDriver.execute() unnecessarily sets METASTOREPWD to HIVE
Hari Sankar Sivarama Subramaniyan created HIVE-12340: Summary: ExecDriver.execute() unnecessarily sets METASTOREPWD to HIVE Key: HIVE-12340 URL: https://issues.apache.org/jira/browse/HIVE-12340 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HIVE-10508/HIVE-11670 covers the issue to mask the hidden user information before doing the MR/Tez job submission.We no longer need to mask the original jdo Connection password with HIVE before submitting the mapred job. conf.stripHiddenConfigurations(job) should do the job of hiding the user password when called from ExecDriver.initialize(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12308) Make ParseContext::semanticInputs a map
Hari Sankar Sivarama Subramaniyan created HIVE-12308: Summary: Make ParseContext::semanticInputs a map Key: HIVE-12308 URL: https://issues.apache.org/jira/browse/HIVE-12308 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Follow up jira for HIVE-7723. Explain plan for complex query with lots of partitions is slow due to in-efficient collection used to find a matching ReadEntity. As part of HIVE-7723, we will create a map during PlanUtils.addPartitionInputs(), we should start with a map if possible in ParseContext::semanticInputs to save CPU burn on this additional operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12084) Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space
Hari Sankar Sivarama Subramaniyan created HIVE-12084:
Summary: Hive queries with ORDER BY and large LIMIT fails with
OutOfMemoryError Java heap space
Key: HIVE-12084
URL: https://issues.apache.org/jira/browse/HIVE-12084
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
STEPS TO REPRODUCE:
{code}
CREATE TABLE `sample_07` ( `code` string , `description` string , `total_emp`
int , `salary` int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS
TextFile;
load data local inpath 'sample_07.csv' into table sample_07;
set hive.limit.pushdown.memory.usage=0.;
select * from sample_07 order by salary LIMIT 9;
{code}
This will result in
{code}
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.hive.ql.exec.TopNHash.initialize(TopNHash.java:113)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initializeOp(ReduceSinkOperator.java:234)
at
org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:68)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
{code}
The basic issue lies with top n optimization. We need a limit for the top n
optimization. Ideally we would detect that the allocated bytes will be bigger
than the "limit.pushdown.memory.usage" without trying to alloc it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11764) Verify the correctness of groupby_cube1.q with MR, Tez and Spark Mode with HIVE-1110
Hari Sankar Sivarama Subramaniyan created HIVE-11764: Summary: Verify the correctness of groupby_cube1.q with MR, Tez and Spark Mode with HIVE-1110 Key: HIVE-11764 URL: https://issues.apache.org/jira/browse/HIVE-11764 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan While Working on HIVE-0, I ran into the following wrong results: https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/spark/groupby_cube1.q.out#L478 This happens in spark mode. The following is the diff. @@ -475,7 +525,6 @@ POSTHOOK: Input: default@t1 3 1 7 1 8 2 -NULL 6 The purpose of this jira is to see why the above query is failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11670) Strip out password information from TezSessionState configuration
Hari Sankar Sivarama Subramaniyan created HIVE-11670: Summary: Strip out password information from TezSessionState configuration Key: HIVE-11670 URL: https://issues.apache.org/jira/browse/HIVE-11670 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Remove password information from configuration copy that is sent to Yarn/Tez. We don't need it there. The config entries can potentially be visible to other users. HIVE-10508 had the fix which removed this in certain places, however, when I initiated a session via Hive Cli, I could still see the password information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11634) Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
Hari Sankar Sivarama Subramaniyan created HIVE-11634: Summary: Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...) Key: HIVE-11634 URL: https://issues.apache.org/jira/browse/HIVE-11634 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11490) Lazily call ASTNode::toStringTree() after tree modification
Hari Sankar Sivarama Subramaniyan created HIVE-11490: Summary: Lazily call ASTNode::toStringTree() after tree modification Key: HIVE-11490 URL: https://issues.apache.org/jira/browse/HIVE-11490 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Currently, we call toStringTree() as part of HIVE-11316 everytime the tree is modified. This is a bad approach as we can lazily delay this to the point when toStringTree() is called again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11491) Lazily call ASTNode::toStringTree() after tree modification
Hari Sankar Sivarama Subramaniyan created HIVE-11491: Summary: Lazily call ASTNode::toStringTree() after tree modification Key: HIVE-11491 URL: https://issues.apache.org/jira/browse/HIVE-11491 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Currently, we call toStringTree() as part of HIVE-11316 everytime the tree is modified. This is a bad approach as we can lazily delay this to the point when toStringTree() is called again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11478) StatsUtils::getWritableSize does not treat non-primitive data types correctly
Hari Sankar Sivarama Subramaniyan created HIVE-11478: Summary: StatsUtils::getWritableSize does not treat non-primitive data types correctly Key: HIVE-11478 URL: https://issues.apache.org/jira/browse/HIVE-11478 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan A simple example would be when you have nested data types. For example, Constant MapConstant Array, Constant Object. Here, we would enter, StatsUtils.getSizeOfMap(), which would in turn execute result += getWritableSize(koi, entry.getKey()); The above call would incorrectly return 0 since the key is a complex data type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11405) Add early termination for recursion in StatsRulesProcFactory$FilterStatsRule.evaluateExpression for OR expression
Hari Sankar Sivarama Subramaniyan created HIVE-11405: Summary: Add early termination for recursion in StatsRulesProcFactory$FilterStatsRule.evaluateExpression for OR expression Key: HIVE-11405 URL: https://issues.apache.org/jira/browse/HIVE-11405 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Prasanth Jayachandran Thanks for [~gopalv] for uncovering this as part of HIVE-11330. Quoting him, The recursion protection works well with an AND expr, but it doesn't work against (OR a=1 (OR a=2 (OR a=3 (OR ...) since the for the rows will never be reduced during recursion due to the nature of the OR. We need to execute a short-circuit to satisfy the OR properly - no case which matches a=1 qualifies for the rest of the filters. Recursion should pass in the numRows - branch1Rows for the branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11315) Compilation performance issues for large queries
Hari Sankar Sivarama Subramaniyan created HIVE-11315: Summary: Compilation performance issues for large queries Key: HIVE-11315 URL: https://issues.apache.org/jira/browse/HIVE-11315 Project: Hive Issue Type: Task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Hive gets occassionally bottlenecked on generating plans for large queries, the majority of the cases time is spent in fetching metadata, partitions and other optimizer transformation related rules. This umbrella jira is intended to track the above mentioned performance issues that have been identified in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11316) Use datastructure that doesnt duplicate any part of string for ASTNode::toStringTree()
Hari Sankar Sivarama Subramaniyan created HIVE-11316: Summary: Use datastructure that doesnt duplicate any part of string for ASTNode::toStringTree() Key: HIVE-11316 URL: https://issues.apache.org/jira/browse/HIVE-11316 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HIVE-11281 uses an approach to memoize toStringTree() for ASTNode. This jira is suppose to alter the string memoization to use a different data structure that doesn't duplicate any part of the string so that we do not run into OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11196) Utilities.getPartitionDesc() Should try to reuse TableDesc object
Hari Sankar Sivarama Subramaniyan created HIVE-11196: Summary: Utilities.getPartitionDesc() Should try to reuse TableDesc object Key: HIVE-11196 URL: https://issues.apache.org/jira/browse/HIVE-11196 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Currently, Utilities.getPartitionDesc() creates a new PartitionDesc object which inturn creates new TableDesc object via Utilities.getTableDesc(part.getTable()) for every call. This value needs to be reused so that we can avoid the expense of creating new Descriptor object wherever possible -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-11141) Improve RuleRegExp when the Expression node stack gets huge
Hari Sankar Sivarama Subramaniyan created HIVE-11141:
Summary: Improve RuleRegExp when the Expression node stack gets
huge
Key: HIVE-11141
URL: https://issues.apache.org/jira/browse/HIVE-11141
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
More and more complex workloads are migrated to Hive from Sql Server, Terradata
etc..
And occasionally Hive gets bottlenecked on generating plans for large queries,
the majority of the cases time is spent in fetching metadata, partitions and
other optimizer transformation related rules
I have attached the query for the test case which needs to be tested after we
setup database as shown below.
{code}
create database dataset_3;
use database dataset_3;
{code}
It seems that the most problematic part of the code as the stack gets arbitrary
long, in RuleRegExp.java
{code}
@Override
public int cost(StackNode stack) throws SemanticException {
int numElems = (stack != null ? stack.size() : 0);
String name = ;
for (int pos = numElems - 1; pos = 0; pos--) {
name = stack.get(pos).getName() + % + name;
Matcher m = pattern.matcher(name);
if (m.matches()) {
return m.group().length();
}
}
return -1;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-11107) Support for Performance regression test suite with TPCDS
Hari Sankar Sivarama Subramaniyan created HIVE-11107: Summary: Support for Performance regression test suite with TPCDS Key: HIVE-11107 URL: https://issues.apache.org/jira/browse/HIVE-11107 Project: Hive Issue Type: Task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Support to add TPCDS queries to the performance regression test suite with Hive CBO turned on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10973) JvmPauseMonitor.incrementMetricsCounter NPE while starting HiveServer2
Hari Sankar Sivarama Subramaniyan created HIVE-10973:
Summary: JvmPauseMonitor.incrementMetricsCounter NPE while
starting HiveServer2
Key: HIVE-10973
URL: https://issues.apache.org/jira/browse/HIVE-10973
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
I build and start HS2 in http mode as follows:
./hive --service hiveserver2 --hiveconf hive.server2.transport.mode=http
--hiveconf hive.root.logger=DEBUG,console --hiveconf
hive.server2.thrift.http.path=cliservice --hiveconf
hive.server2.thrift.port=10001
I am hitting a Null Pointer Exception around line 203 as follows:
{code}
15/06/09 13:46:01
[org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@5d648bfd]: WARN
common.JvmPauseMonitor: Error Reporting JvmPauseMonitor to Metrics system
java.lang.NullPointerException
at
org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.incrementMetricsCounter(JvmPauseMonitor.java:203)
at
org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:195)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10949) Disable hive-minikdc tests in Windows
Hari Sankar Sivarama Subramaniyan created HIVE-10949: Summary: Disable hive-minikdc tests in Windows Key: HIVE-10949 URL: https://issues.apache.org/jira/browse/HIVE-10949 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan hive-minikdc needs to be disabled for Windows OS since we dont have kerberos support yet for Hadoop Cluster running under Windows OS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10941) Provide option to disable spark tests outside itests
Hari Sankar Sivarama Subramaniyan created HIVE-10941:
Summary: Provide option to disable spark tests outside itests
Key: HIVE-10941
URL: https://issues.apache.org/jira/browse/HIVE-10941
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
HIVE-10477 provided an option to disable spark module, however we missed the
following files that are outside itests directory. i.e we need to club the
option with disabling the following tests as well :
{code}
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
The above tests need to be disabled.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10919) Windows: create table with JsonSerDe failed via beeline unless you add hcatalog core jar to classpath
Hari Sankar Sivarama Subramaniyan created HIVE-10919:
Summary: Windows: create table with JsonSerDe failed via beeline
unless you add hcatalog core jar to classpath
Key: HIVE-10919
URL: https://issues.apache.org/jira/browse/HIVE-10919
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Before we run HiveServer2 tests, we create table via beeline.
And 'create table' with JsonSerDe failed on Winodws. It works on Linux:
{noformat}
0: jdbc:hive2://localhost:10001 create external table all100kjson(
0: jdbc:hive2://localhost:10001 s string,
0: jdbc:hive2://localhost:10001 i int,
0: jdbc:hive2://localhost:10001 d double,
0: jdbc:hive2://localhost:10001 m mapstring, string,
0: jdbc:hive2://localhost:10001 bb arraystructa: int, b: string,
0: jdbc:hive2://localhost:10001 t timestamp)
0: jdbc:hive2://localhost:10001 row format serde
'org.apache.hive.hcatalog.data.JsonSerDe'
0: jdbc:hive2://localhost:10001 WITH SERDEPROPERTIES
('timestamp.formats'='-MM-dd\'T\'HH:mm:ss')
0: jdbc:hive2://localhost:10001 STORED AS TEXTFILE
0: jdbc:hive2://localhost:10001 location '/user/hcat/tests/data/all100kjson';
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.DDLT
ask. Cannot validate serde: org.apache.hive.hcatalog.data.JsonSerDe
(state=08S01,code=1)
{noformat}
hive.log shows:
{noformat}
2015-05-21 21:59:17,004 ERROR operation.Operation (SQLOperation.java:run(209))
- Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde:
org.apache.hive.hcatalog.data.JsonSerDe
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Cannot validate
serde: org.apache.hive.hcatalog.data.JsonSerDe
at
org.apache.hadoop.hive.ql.exec.DDLTask.validateSerDe(DDLTask.java:3871)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4011)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
... 11 more
Caused by: java.lang.ClassNotFoundException: Class
org.apache.hive.hcatalog.data.JsonSerDe not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at
org.apache.hadoop.hive.ql.exec.DDLTask.validateSerDe(DDLTask.java:3865)
... 21 more
{noformat}
If you do add the hcatalog jar to classpath, it works:
{noformat}0: jdbc:hive2://localhost:10001 add jar
hdfs:///tmp/testjars/hive-hcatalog-core-1.2.0.2.3.0.0-2079.jar;
INFO : converting to local
hdfs:///tmp/testjars/hive-hcatalog-core-1.2.0.2.3.0.0-2079.jar
INFO : Added
[/C:/Users/hadoop/AppData/Local/Temp/bc941dac-3bca-4287-a490-8a65c2dac220_resources/hive-hcatalog-core-1.2
.0.2.3.0.0-2079.jar] to class path
INFO : Added resources:
[hdfs:///tmp/testjars/hive-hcatalog-core-1.2.0.2.3.0.0-2079.jar]
No rows affected (0.304 seconds)
0: jdbc:hive2://localhost:10001 create external table all100kjson(
0: jdbc:hive2://localhost:10001 s string,
0:
[jira] [Created] (HIVE-10887) TestCliDriver tests ordering issues with Mac and CentOS
Hari Sankar Sivarama Subramaniyan created HIVE-10887: Summary: TestCliDriver tests ordering issues with Mac and CentOS Key: HIVE-10887 URL: https://issues.apache.org/jira/browse/HIVE-10887 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan unionDistinct_2 and update_after_multiple_inserts tests give different results in different environments due to ordering issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10865) Beeline needs to support DELIMITER command
Hari Sankar Sivarama Subramaniyan created HIVE-10865: Summary: Beeline needs to support DELIMITER command Key: HIVE-10865 URL: https://issues.apache.org/jira/browse/HIVE-10865 Project: Hive Issue Type: Bug Components: Beeline Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan MySQL Client provides a DELIMITER command to set statement delimiter. Beeline needs to support a similar command to allow commands having semi-colon as non-statement delimiter (as with MySQL stored procedures). This is a follow-up jira for HIVE-10659 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10839) TestHCatLoaderEncryption.* tests fail in windows because of path related issues
Hari Sankar Sivarama Subramaniyan created HIVE-10839:
Summary: TestHCatLoaderEncryption.* tests fail in windows because
of path related issues
Key: HIVE-10839
URL: https://issues.apache.org/jira/browse/HIVE-10839
Project: Hive
Issue Type: Bug
Components: Tests
Environment: Windows OS
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
I am getting the following errors while trying to run
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* tests in windows.
{code}
Encryption key created: 'key_128'
(1,Encryption Processor Helper Failed:Pathname
/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1432579852919/warehouse/encryptedTable
from
D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1432579852919/warehouse/encryptedTable
is not a valid DFS filename.,null)
Encryption key deleted: 'key_128'
{code}
{code}
Error Message
Could not fully delete
D:\w\hv\hcatalog\hcatalog-pig-adapter\target\tmp\dfs\name1
Stacktrace
java.io.IOException: Could not fully delete
D:\w\hv\hcatalog\hcatalog-pig-adapter\target\tmp\dfs\name1
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:940)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:811)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:742)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:612)
at
org.apache.hadoop.hive.shims.Hadoop23Shims.getMiniDfs(Hadoop23Shims.java:523)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.initEncryptionShim(TestHCatLoaderEncryption.java:242)
at
org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:190)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10801) 'drop view' fails throwing java.lang.NullPointerException
Hari Sankar Sivarama Subramaniyan created HIVE-10801:
Summary: 'drop view' fails throwing java.lang.NullPointerException
Key: HIVE-10801
URL: https://issues.apache.org/jira/browse/HIVE-10801
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
When trying to drop a view, hive log shows:
{code}
2015-05-21 11:53:06,126 ERROR [HiveServer2-Background-Pool: Thread-197]:
hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) - Could
not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider
!!
2015-05-21 11:53:06,134 ERROR [HiveServer2-Background-Pool: Thread-197]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invoke(155)) -
MetaException(message:java.lang.NullPointerException)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:5379)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1734)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy7.drop_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.drop_table_with_environment_context(HiveMetaStoreClient.java:2056)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.drop_table_with_environment_context(SessionHiveMetaStoreClient.java:118)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:968)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropTable(HiveMetaStoreClient.java:904)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy8.dropTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:1035)
at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:972)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3836)
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3692)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:331)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1650)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1409)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1192)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1213)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1546)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1723)
... 40 more
[jira] [Created] (HIVE-10768) In QTestGenTask.execute() we should not throw an exception right away if we are unable to clean any old files
Hari Sankar Sivarama Subramaniyan created HIVE-10768:
Summary: In QTestGenTask.execute() we should not throw an
exception right away if we are unable to clean any old files
Key: HIVE-10768
URL: https://issues.apache.org/jira/browse/HIVE-10768
Project: Hive
Issue Type: Bug
Components: Testing Infrastructure
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Currently in QTestGenTask.execute(), we have code like :
{code}
if (logFile != null) {
File lf = new File(logFile);
if (lf.exists()) {
if (!lf.delete()) {
throw new Exception(Could not delete log file +
lf.getCanonicalPath());
}
{code}
If the log file gets locked due to some other process, it throws the above
exception and short circuits the entire test suite. We should provide a
mechanism to generate alternative new log files/temporary files whenever
possible to avoid unit test failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10753) hs2 jdbc url - wrong connection string cause OOM error on beeline/jdbc/odbc client, misleading message
Hari Sankar Sivarama Subramaniyan created HIVE-10753:
Summary: hs2 jdbc url - wrong connection string cause OOM error on
beeline/jdbc/odbc client, misleading message
Key: HIVE-10753
URL: https://issues.apache.org/jira/browse/HIVE-10753
Project: Hive
Issue Type: Bug
Components: Beeline, JDBC
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{noformat}
beeline -u 'jdbc:hive2://localhost:10001/default?httpPath=/;transportMode=http'
-n hdiuser
scan complete in 15ms
Connecting to jdbc:hive2://localhost:10001/default?httpPath=/;transportMode=http
Java heap space
Beeline version 0.14.0.2.2.4.1-1 by Apache Hive
0: jdbc:hive2://localhost:10001/default (closed) ^Chdiuser@headnode0:~$
But it works if I use the deprecated param -
hdiuser@headnode0:~$ beeline -u
'jdbc:hive2://localhost:10001/default?hive.server2.transport.mode=http;httpPath=/'
-n hdiuser
scan complete in 12ms
Connecting to
jdbc:hive2://localhost:10001/default?hive.server2.transport.mode=http;httpPath=/
15/04/28 23:16:46 [main]: WARN jdbc.Utils: * JDBC param deprecation *
15/04/28 23:16:46 [main]: WARN jdbc.Utils: The use of
hive.server2.transport.mode is deprecated.
15/04/28 23:16:46 [main]: WARN jdbc.Utils: Please use transportMode like so:
jdbc:hive2://host:port/dbName;transportMode=transport_mode_value
Connected to: Apache Hive (version 0.14.0.2.2.4.1-1)
Driver: Hive JDBC (version 0.14.0.2.2.4.1-1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.14.0.2.2.4.1-1 by Apache Hive
0: jdbc:hive2://localhost:10001/default show tables;
+--+--+
| tab_name |
+--+--+
| hivesampletable |
+--+--+
1 row selected (18.181 seconds)
0: jdbc:hive2://localhost:10001/default ^Chdiuser@headnode0:~$ ^C
{noformat}
The reason for the above message is :
The url is wrong. Correct one:
{code}
beeline -u 'jdbc:hive2://localhost:10001/default;httpPath=/;transportMode=http'
-n hdiuser
{code}
Note the ; instead of ?. The deprecation msg prints the format as well:
{code}
Please use transportMode like so:
jdbc:hive2://host:port/dbName;transportMode=transport_mode_value
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10696) TestAddResource tests are non-portable
Hari Sankar Sivarama Subramaniyan created HIVE-10696: Summary: TestAddResource tests are non-portable Key: HIVE-10696 URL: https://issues.apache.org/jira/browse/HIVE-10696 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We need to make sure these tests work in windows as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10670) Duplicate declaration of curator-recipes at pom.xml
Hari Sankar Sivarama Subramaniyan created HIVE-10670:
Summary: Duplicate declaration of curator-recipes at pom.xml
Key: HIVE-10670
URL: https://issues.apache.org/jira/browse/HIVE-10670
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Unnecessary duplicate declaration of curator-recipes is causing warnings with
the build.
{code}
dependency
groupIdorg.apache.curator/groupId
artifactIdcurator-recipes/artifactId
version${curator.version}/version
/dependency
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10675) Provide option to skip Accumulo related Hive tests in itests directory
Hari Sankar Sivarama Subramaniyan created HIVE-10675: Summary: Provide option to skip Accumulo related Hive tests in itests directory Key: HIVE-10675 URL: https://issues.apache.org/jira/browse/HIVE-10675 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We shouldn’t be running Accumulo+Hive tests in Windows as Accumulo is currently unsupported on Windows. Hence we should provide an option to disable these tests; that being mentioned, the default behavior would be to enable these tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10664) Unit tests run fail in windows because of illegal escape character in file path
Hari Sankar Sivarama Subramaniyan created HIVE-10664:
Summary: Unit tests run fail in windows because of illegal escape
character in file path
Key: HIVE-10664
URL: https://issues.apache.org/jira/browse/HIVE-10664
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
{code:title=In windows we hit errors as shown below }
[ERROR]
/D:/w/hv/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestHBaseNegativeCliDriver.java:[97,54]
illegal escape character
{code}
Specifically, the lines it is complaining about in the log, look like this:
{code:title=line 97 of
/itests/qtest/target/generated-test-sources/java/org/apache/hadoop/hive/cli/TestHBaseNegativeCliDriver.java}
line 97:
QTestUtil.addTestsToSuiteFromQfileNames(D:\w\hv\itests\qtest\target\generated-test-sources\java\org\apache\hadoop\hive\cli\TestHBaseNegativeCliDriverQFileNames.txt,
qFilesToExecute,
{code}
It is executing the itests/qtests directory on the windows platform. It seems
as if it is hitting a fairly simple portability problem, of the type that a
small change will touch up all of them quickly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10659) Beeline commands which contains semi-colon as a non-command terminator will fail
Hari Sankar Sivarama Subramaniyan created HIVE-10659: Summary: Beeline commands which contains semi-colon as a non-command terminator will fail Key: HIVE-10659 URL: https://issues.apache.org/jira/browse/HIVE-10659 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Consider beeline for connecting to mysql and creating commands involving stored procedures. MySQL stored procedures have semi-colon as the statement terminator. Since this coincides with beeline's only available command terminator , semi-colon, beeline will not able to execute the original command successfully. The above scenario can happen when Hive SchemaTool is used to upgrade a mysql metastore db which contains stored procedure in the script(as the one introduced initially by HIVE-7018). As of now, we cannot have any stored procedures as part of MySQL scripts because schemaTool uses beeline as the jdbc client to connect to MySQL. This is a serious limitation and needs to be fixed by providing an option to beeline to not use ; as the command delimiter and process the entire line send to it as a single command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10635) Redo HIVE-7018 in a schematool compatible manner
Hari Sankar Sivarama Subramaniyan created HIVE-10635: Summary: Redo HIVE-7018 in a schematool compatible manner Key: HIVE-10635 URL: https://issues.apache.org/jira/browse/HIVE-10635 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan In HIVE-10614, we had to revert HIVE-7018 because it was not schematool compatible and it would prevent upgrade from 0.14.0 to 1.3.0 when run via schematool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10614) schemaTool upgrade from 0.14.0 to 1.3.0 causes failure
Hari Sankar Sivarama Subramaniyan created HIVE-10614:
Summary: schemaTool upgrade from 0.14.0 to 1.3.0 causes failure
Key: HIVE-10614
URL: https://issues.apache.org/jira/browse/HIVE-10614
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Priority: Critical
./schematool -dbType mysql -upgradeSchemaFrom 0.14.0 -verbose
{code}
++--+
|
|
++--+
| HIVE-7018 Remove Table and Partition tables column LINK_TARGET_ID from
Mysql for other DBs do not have it |
++--+
1 row selected (0.004 seconds)
0: jdbc:mysql://node-1.example.com/hive DROP PROCEDURE IF EXISTS RM_TLBS_LINKID
No rows affected (0.005 seconds)
0: jdbc:mysql://node-1.example.com/hive DROP PROCEDURE IF EXISTS
RM_PARTITIONS_LINKID
No rows affected (0.006 seconds)
0: jdbc:mysql://node-1.example.com/hive DROP PROCEDURE IF EXISTS RM_LINKID
No rows affected (0.002 seconds)
0: jdbc:mysql://node-1.example.com/hive CREATE PROCEDURE RM_TLBS_LINKID()
BEGIN IF EXISTS (SELECT * FROM `INFORMATION_SCHEMA`.`COLUMNS` WHERE
`TABLE_NAME` = 'TBLS' AND `COLUMN_NAME` = 'LINK_TARGET_ID') THEN ALTER TABLE
`TBLS` DROP FOREIGN KEY `TBLS_FK3` ; ALTER TABLE `TBLS` DROP KEY `TBLS_N51` ;
ALTER TABLE `TBLS` DROP COLUMN `LINK_TARGET_ID` ; END IF; END
Error: You have an error in your SQL syntax; check the manual that corresponds
to your MySQL server version for the right syntax to use near '' at line 1
(state=42000,code=1064)
Closing: 0: jdbc:mysql://node-1.example.com/hive?createDatabaseIfNotExist=true
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore
state would be inconsistent !!
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:229)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:468)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:355)
at
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:326)
at
org.apache.hive.beeline.HiveSchemaTool.doUpgrade(HiveSchemaTool.java:224)
{code}
Looks like HIVE-7018 has introduced stored procedure as part of mysql upgrade
script and it is causing issues with schematool upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10576) add jar command is not portable in non-unix systems
Hari Sankar Sivarama Subramaniyan created HIVE-10576:
Summary: add jar command is not portable in non-unix systems
Key: HIVE-10576
URL: https://issues.apache.org/jira/browse/HIVE-10576
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
Steps to reproduce this issue in Windows OS:
hadoop.cmd fs -mkdir -p /tmp/testjars
hadoop.cmd fs -copyFromLocal hive-hcatalog-core-*.jar /tmp/testjars
from hive cli:
add jar hdfs:///tmp/testjars/hive-hcatalog-core-*.jar;
add jar D:\hdp\hive-1.2.0.2.3.0.0-1737\hcatalog\share\hcatalog\hive-hcatal
og-core-1.2.0.2.3.0.0-1737.jar;
{code}
hive add jar hdfs:///tmp/testjars/hive-hcatalog-core-1.2.0.2.3.0.0-1737.jar;
converting to local hdfs:///tmp/testjars/hive-hcatalog-core-1.2.0.2.3.0.0-1737.j
ar
Illegal character in opaque part at index 2: C:\Users\hadoopqa\AppData\Local\Tem
p\cf0c70a4-f8e5-43ae-8c94-aa528f90887d_resources\hive-hcatalog-core-1.2.0.2.3.0.
0-1737.jar
Query returned non-zero code: 1, cause: java.net.URISyntaxException: Illegal cha
racter in opaque part at index 2: C:\Users\hadoopqa\AppData\Local\Temp\cf0c70a4-
f8e5-43ae-8c94-aa528f90887d_resources\hive-hcatalog-core-1.2.0.2.3.0.0-1737.jar
hive add jar D:\hdp\hive-1.2.0.2.3.0.0-1737\hcatalog\share\hcatalog\hive-hcatal
og-core-1.2.0.2.3.0.0-1737.jar;
Illegal character in opaque part at index 2: D:\hdp\hive-1.2.0.2.3.0.0-1737\hcat
alog\share\hcatalog\hive-hcatalog-core-1.2.0.2.3.0.0-1737.jar
Query returned non-zero code: 1, cause: java.net.URISyntaxException: Illegal cha
racter in opaque part at index 2: D:\hdp\hive-1.2.0.2.3.0.0-1737\hcatalog\share\
hcatalog\hive-hcatalog-core-1.2.0.2.3.0.0-1737.jar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10563) MiniTezCliDriver tests ordering issues
Hari Sankar Sivarama Subramaniyan created HIVE-10563: Summary: MiniTezCliDriver tests ordering issues Key: HIVE-10563 URL: https://issues.apache.org/jira/browse/HIVE-10563 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan There are a bunch of tests related to TestMiniTezCliDriver which gives ordering issues when run on Centos/Windows/OSX -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10544) Beeline/Hive JDBC Driver fails in HTTP mode on Windows with java.lang.NoSuchFieldError: INSTANCE
Hari Sankar Sivarama Subramaniyan created HIVE-10544: Summary: Beeline/Hive JDBC Driver fails in HTTP mode on Windows with java.lang.NoSuchFieldError: INSTANCE Key: HIVE-10544 URL: https://issues.apache.org/jira/browse/HIVE-10544 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan This appears to be caused by a dependency version mispatch with httpcore on Beeline's classpath. We need to change beeline.cmd as well I guess to include the equivalent of export HADOOP_USER_CLASSPATH_FIRST=true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10507) Expose RetryingMetastoreClient to other external users of metastore client like Flume and Storm.
Hari Sankar Sivarama Subramaniyan created HIVE-10507: Summary: Expose RetryingMetastoreClient to other external users of metastore client like Flume and Storm. Key: HIVE-10507 URL: https://issues.apache.org/jira/browse/HIVE-10507 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HiveMetastoreClient is now being relied upon by external clients like Flume and Storm for streaming. When the thrift connection between MetaStoreClient and the meta store is broken (due to intermittent network issues or restarting of metastore) the Metastore does not handle the connection error and automatically re-establish the connection. Currently the client process needs to be restarted to re-establish the connection. The request here is consider supporting the following behavior: For each API invocation on the MetastoreClient, it should try to restablish the connection (if needed) once. And if that does not work out then throw a specific exception indicating the same. The client could then handle the issue by retrying the same API after some delay. By catching the specific connection exception, the client could decide how many times to retry before aborting. Hive does this internally using RetryingMetastoreClient. This jira is suppose to expose this mechanism to other users of that interface. This is useful for users of this interface, and from metastore HA point of view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10508) Strip out password information from config passed to Tez/MR in cases where password encryption is not used
Hari Sankar Sivarama Subramaniyan created HIVE-10508: Summary: Strip out password information from config passed to Tez/MR in cases where password encryption is not used Key: HIVE-10508 URL: https://issues.apache.org/jira/browse/HIVE-10508 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Remove password information from configuration copy that is sent to Yarn/Tez. We don't need it there. The config entries can potentially be visible to other users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10477) Provide option to disable Spark tests in Windows OS
Hari Sankar Sivarama Subramaniyan created HIVE-10477:
Summary: Provide option to disable Spark tests in Windows OS
Key: HIVE-10477
URL: https://issues.apache.org/jira/browse/HIVE-10477
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
In the current master branch, unit tests fail with windows OS because of the
dependency on bash executable in itests/hive-unit/pom.xml around these lines :
{code}
target
exec executable=bash dir=${basedir} failonerror=true
arg line=../target/download.sh/
/exec
/target
{code}
We should provide an option to disable spark tests in OSes like Windows where
bash might be absent
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10447) Beeline JDBC Driver to support 2 way SSL
Hari Sankar Sivarama Subramaniyan created HIVE-10447: Summary: Beeline JDBC Driver to support 2 way SSL Key: HIVE-10447 URL: https://issues.apache.org/jira/browse/HIVE-10447 Project: Hive Issue Type: Bug Components: Beeline Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan This jira should cover 2-way SSL authentication between the JDBC Client and server which requires the driver to support it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10432) Need to add more e2e like tests between HiveServer2 and JDBC using wiremock or equivalent
Hari Sankar Sivarama Subramaniyan created HIVE-10432: Summary: Need to add more e2e like tests between HiveServer2 and JDBC using wiremock or equivalent Key: HIVE-10432 URL: https://issues.apache.org/jira/browse/HIVE-10432 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan The current unit tests use ThriftCLIService to test client-server interaction. We will need to mock HS2 to facilitate use of writing test cases where we can parse HTTP request/response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10367) org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows
Hari Sankar Sivarama Subramaniyan created HIVE-10367: Summary: org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.* fails in Windows Key: HIVE-10367 URL: https://issues.apache.org/jira/browse/HIVE-10367 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan java.io.IOException: Failed to execute create table encrypted_table(a int, b string) location 'D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable'. Driver returned 1 Error: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: Pathname /D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable from hdfs://127.0.0.1:64735/D:/w/hv/hcatalog/hcatalog-pig-adapter/target/tmp/org.apache.hive.hcatalog.pig.TestHCatLoader-1429172673960/warehouse/encryptedTable is not a valid DFS filename.) at org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.executeStatementOnDriver(TestHCatLoaderEncryption.java:163) at org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.createTableInSpecifiedPath(TestHCatLoaderEncryption.java:423) at org.apache.hive.hcatalog.pig.TestHCatLoaderEncryption.setup(TestHCatLoaderEncryption.java:197) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10370) Hive does not compile with -Phadoop-1 option
Hari Sankar Sivarama Subramaniyan created HIVE-10370:
Summary: Hive does not compile with -Phadoop-1 option
Key: HIVE-10370
URL: https://issues.apache.org/jira/browse/HIVE-10370
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Priority: Critical
Running into the below error while running mvn clean install -Pdist -Phadoop-1
{code}
[ERROR]hive/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleFast.java:[164,33]
cannot find symbol
symbol: method copyBytes()
location: variable serialized of type org.apache.hadoop.io.Text
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10345) Add test case to ensure client sends credentials in non-ssl mode when HS2 sends a secure cookie
Hari Sankar Sivarama Subramaniyan created HIVE-10345: Summary: Add test case to ensure client sends credentials in non-ssl mode when HS2 sends a secure cookie Key: HIVE-10345 URL: https://issues.apache.org/jira/browse/HIVE-10345 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We need to add test cases to cover these scenarios. _ Client | HS2 Cookie | Expected Behavior ___| _ |___ SSL | Secured | Client replays, server validates the cookie. SSL | Unsecured| Client replays, server validates the cookie. No SSL | UnSecured| Client replays, server validates the cookie. No SSL | Secured | Client should send back credentials since cookie | | replay will not be transmitted back to the server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10339) Allow JDBC Driver to pass HTTP header Key/Value pairs
Hari Sankar Sivarama Subramaniyan created HIVE-10339: Summary: Allow JDBC Driver to pass HTTP header Key/Value pairs Key: HIVE-10339 URL: https://issues.apache.org/jira/browse/HIVE-10339 Project: Hive Issue Type: Bug Components: Beeline Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Currently Beeline ODBC driver does not support carrying user specified HTTP header. The beeline JDBC driver in HTTP mode connection string is as jdbc:hive2://host:port/db?hive.server2.transport.mode=http;hive.server2.thrift.http.path=http_endpoint, When transport mode is http Beeline/ODBC driver should allow end user to send arbitrary HTTP Header name value pair. All the beeline driver needs to do is to use the user specified name values and call the underlying HTTPClient API to set the header. E.g the Beeline connection string could be jdbc:hive2://host:port/db?hive.server2.transport.mode=http;hive.server2.thrift.http.path=http_endpoint,http.header.name1=value1, And the beeline will call underlying to set HTTP header to name1 and value1 This is required for the end user to send identity in a HTTP header down to Knox via beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10309) TestJdbcWithMiniHS2.java broken because of the removal of hive.server2.thrift.http.max.worker.threads
Hari Sankar Sivarama Subramaniyan created HIVE-10309: Summary: TestJdbcWithMiniHS2.java broken because of the removal of hive.server2.thrift.http.max.worker.threads Key: HIVE-10309 URL: https://issues.apache.org/jira/browse/HIVE-10309 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan HIVE-10271 removed hive.server2.thrift.http.min/max.worker.threads properties, however these properties are used in a few more places in hive code. For example, TestJdbcWithMiniHS2.java and hive/conf/hive-default.xml.template. We need to fix these as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10306) We need to print tez summary when hive.server2.logging.level = PERFORMANCE.
Hari Sankar Sivarama Subramaniyan created HIVE-10306: Summary: We need to print tez summary when hive.server2.logging.level = PERFORMANCE. Key: HIVE-10306 URL: https://issues.apache.org/jira/browse/HIVE-10306 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We need to print tez summary when hive.server2.logging.level = PERFORMANCE. We introduced this parameter via HIVE-10119. The logging param for levels is only relevant to HS2, so for hive-cli users the hive.tez.exec.print.summary still makes sense. We can check for log-level param as well, in places we are checking value of hive.tez.exec.print.summary. Ie, consider hive.tez.exec.print.summary=true if log.level = PERFORMANCE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10271) remove hive.server2.thrift.http.min/max.worker.threads properties
Hari Sankar Sivarama Subramaniyan created HIVE-10271: Summary: remove hive.server2.thrift.http.min/max.worker.threads properties Key: HIVE-10271 URL: https://issues.apache.org/jira/browse/HIVE-10271 Project: Hive Issue Type: Bug Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan PROBLEM: Those properties are not used even when hiveserver2 in http mode. The properties used are hive.server2.thrift.min/max.worker.threads. Remove those 2 properties as they are causing confusion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10119) Allow Log verbosity to be set in hiveserver2 session
Hari Sankar Sivarama Subramaniyan created HIVE-10119: Summary: Allow Log verbosity to be set in hiveserver2 session Key: HIVE-10119 URL: https://issues.apache.org/jira/browse/HIVE-10119 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan We need to be able to set logging per HS2 session. The client often uses the map-reduce completion matrix (Execution) that shows up in Beeline to debug performance. User might not want the verbose log view all the time since it obfuscates the Execution information. Hence the client should be able to change the verbosity level. Also, there are 2 levels of verbosity at HS2 logging and not 3. The users might want Execution + Performance counters only - so that level needs to be added. So for logs, the user should be able to set 3 levels of verbosity in the session, that will override the default verbosity specified in the hive-site.xml file. 0. None - IGNORE 1. Execution - Just shows the map-reduce tasks completing 2. Performance - Execution + Performance counters dumped at the end 3. Verbose - All logs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10061) HiveConf Should not be used as part of the HS2 client side code
Hari Sankar Sivarama Subramaniyan created HIVE-10061:
Summary: HiveConf Should not be used as part of the HS2 client
side code
Key: HIVE-10061
URL: https://issues.apache.org/jira/browse/HIVE-10061
Project: Hive
Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
HiveConf crept in to the JDBC driver via the embedded mode check.
if (isEmbeddedMode) {
EmbeddedThriftBinaryCLIService embeddedClient = new
EmbeddedThriftBinaryCLIService();
embeddedClient.init(new HiveConf());
client = embeddedClient;
} else {
Ideally we'd like to keep driver code free of these dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9731) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified
Hari Sankar Sivarama Subramaniyan created HIVE-9731: --- Summary: WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified Key: HIVE-9731 URL: https://issues.apache.org/jira/browse/HIVE-9731 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Hadoop Streaming allows -inputreader parameter to specify use of StreamXmlRecorderReader (example) hadoop jar hadoop-streaming-2.5.1.jar \ -inputreader StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING \ (rest of the command) WebHCat's StreamingDelegator does not include -inputreader as a valid option when submitting jobs to http://www.myserver.com/templeton/v1/mapreduce/streaming endpoint. If -inputreader is specified and passed to templeton server (perhaps via CURL operation) , it will get truncated and not passed as parameter from TempletonControllerJob to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9731) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified
[ https://issues.apache.org/jira/browse/HIVE-9731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-9731: Status: Patch Available (was: Open) WebHCat MapReduce Streaming Job does not allow StreamXmlRecordReader to be specified Key: HIVE-9731 URL: https://issues.apache.org/jira/browse/HIVE-9731 Project: Hive Issue Type: Bug Components: WebHCat Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan Attachments: HIVE-9731.1.patch Hadoop Streaming allows -inputreader parameter to specify use of StreamXmlRecorderReader (example) hadoop jar hadoop-streaming-2.5.1.jar \ -inputreader StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING \ (rest of the command) WebHCat's StreamingDelegator does not include -inputreader as a valid option when submitting jobs to http://www.myserver.com/templeton/v1/mapreduce/streaming endpoint. If -inputreader is specified and passed to templeton server (perhaps via CURL operation) , it will get truncated and not passed as parameter from TempletonControllerJob to Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
