[jira] [Created] (HIVE-9752) Documentation for HBase metastore
Alan Gates created HIVE-9752: Summary: Documentation for HBase metastore Key: HIVE-9752 URL: https://issues.apache.org/jira/browse/HIVE-9752 Project: Hive Issue Type: Sub-task Components: Documentation Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates All of the documentation we will need to write for the HBase metastore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Empty result pb
Hi, I'm facing a particularly annoying bug with Beeline and CLI. I'm using Hadoop 2.5.1 and Hive 1.0.0 (Tez 0.5.3) without any pb. But I'm currently testing going to Hadoop 2.6.0. I testing Hadoop 2.6.0 with combination of Hive 1.1/1.2-SNAPSHOT and Tez 0.5.3/Tez0.6.0. Here is my problem : Every query I make on Hadoop 2.6.0, I'm getting an empty result. If the result is a simple Fetch, I can see data. But, if Tez is used in the query I have this problem. For example, if I do this : SELECT id, count(*) as nb FROM bar GROUP BY id = empty result, I can see the right headers (id, _c1) but if I do this : CREATE TABLE foo AS SELECT id, count(*) as nb FROM bar GROUP BY id I can see data in foo table and a simple select in foo table give me the right result. I crawled user mailing list without success. I tried to restart all nodes in the cluster, removing all jars in HDFS without success. I see no solutions. Any tips on my problem ? Or what should I check in configuration? Regards, Damien CAROL - tél : +33 (0)4 74 96 88 14 - email : dca...@blitzbs.com BLITZ BUSINESS SERVICE
[jira] [Created] (HIVE-9755) Hive built-in ngram UDAF fails when a mapper has no matches.
Naveen Gangam created HIVE-9755: --- Summary: Hive built-in ngram UDAF fails when a mapper has no matches. Key: HIVE-9755 URL: https://issues.apache.org/jira/browse/HIVE-9755 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.14.0 Reporter: Naveen Gangam Assignee: Naveen Gangam Priority: Critical -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9753) Wrong results when using multiple levels of Joins. When table alias of one of the table is null with left outer joins.
Pavan Srinivas created HIVE-9753:
Summary: Wrong results when using multiple levels of Joins. When
table alias of one of the table is null with left outer joins.
Key: HIVE-9753
URL: https://issues.apache.org/jira/browse/HIVE-9753
Project: Hive
Issue Type: Bug
Reporter: Pavan Srinivas
Priority: Critical
Let take scenario, where the tables are:
{code}
drop table table1;
CREATE TABLE table1(
col1 string,
col2 string,
col3 string,
col4 string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
drop table table2;
CREATE TABLE table2(
col1 string,
col2 bigint,
col3 string,
col4 string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
drop table table3;
CREATE TABLE table3(
col1 string,
col2 int,
col3 int,
col4 string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat';
{code}
Query with wrong results:
{code}
SELECT t1.col1 AS dummy,
t1.expected_column AS expected_column,
t2.col4
FROM (
SELECT col1,
'23-1',
'23-13' as three,
col4 AS expected_column
FROM table1
) t1
JOIN table2 t2
ON cast(t2.col1 as string) = cast(t1.col1 as string)
LEFT OUTER JOIN
(SELECT col4, col1
FROM table3
) t3
ON t2.col4 = t3.col1
;
{code}
and explain output:
{code}
STAGE DEPENDENCIES:
Stage-7 is a root stage
Stage-5 depends on stages: Stage-7
Stage-0 depends on stages: Stage-5
STAGE PLANS:
Stage: Stage-7
Map Reduce Local Work
Alias - Map Local Tables:
t1:table1
Fetch Operator
limit: -1
t3:table3
Fetch Operator
limit: -1
Alias - Map Local Operator Tree:
t1:table1
TableScan
alias: table1
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Filter Operator
predicate: col1 is not null (type: boolean)
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Select Operator
expressions: col1 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
HashTable Sink Operator
condition expressions:
0
1 {col4}
keys:
0 _col0 (type: string)
1 col1 (type: string)
t3:table3
TableScan
alias: table3
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Select Operator
expressions: col1 (type: string)
outputColumnNames: _col1
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
HashTable Sink Operator
condition expressions:
0 {_col0} {_col7} {_col7}
1
keys:
0 _col7 (type: string)
1 _col1 (type: string)
Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
alias: t2
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Filter Operator
predicate: col1 is not null (type: boolean)
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {_col0}
1 {col4}
keys:
0 _col0 (type: string)
1 col1 (type: string)
outputColumnNames: _col0, _col7
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Map Join Operator
condition map:
Left Outer Join0 to 1
condition expressions:
0 {_col0} {_col7} {_col7}
1
keys:
0 _col7 (type: string)
1 _col1 (type: string)
outputColumnNames: _col0, _col3, _col7
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
Select Operator
expressions: _col0 (type: string), _col3 (type: string),
_col7 (type: string)
Review Request 31316: HIVE-9754 rename GenericUDFLevenstein to GenericUDFLevenshtein
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31316/ --- Review request for hive and Jason Dere. Bugs: HIVE-9678 https://issues.apache.org/jira/browse/HIVE-9678 Repository: hive-git Description --- HIVE-9754 rename GenericUDFLevenstein to GenericUDFLevenshtein Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java bfeb33c9edf79071bf106a3780e1c6bc21be3753 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLevenstein.java de41793ba3925aa9e1ad9623d92881c57791f047 ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLevenshtein.java e674d9f38cf7b5cdffcad6eca07dba74ff1e834b Diff: https://reviews.apache.org/r/31316/diff/ Testing --- Thanks, Alexander Pivovarov
[jira] [Created] (HIVE-9754) rename GenericUDFLevenstein to GenericUDFLevenshtein
Alexander Pivovarov created HIVE-9754: - Summary: rename GenericUDFLevenstein to GenericUDFLevenshtein Key: HIVE-9754 URL: https://issues.apache.org/jira/browse/HIVE-9754 Project: Hive Issue Type: Bug Components: UDF Reporter: Alexander Pivovarov Assignee: Alexander Pivovarov Priority: Trivial GenericUDFLevenstein has typo (s instead of sh) It should be GenericUDFLevenshtein -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31249/ --- (Updated Feb. 23, 2015, 10:08 p.m.) Review request for hive. Changes --- Two points: 1. Using getProductName based JDBC driver to get dbType info. 2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf. Summary (updated) - Refactor MetaStoreDirectSql constructor Repository: hive-git Description (updated) --- MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked. Two points: 1. Using getProductName based JDBC driver to get dbType info. 2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf. Here’s stack trace: MetaStoreDirectSql.determineDbType(...) MetaStoreDirectSql.MetaStoreDirectSql(...) ObjectStore.initialize(...) ObjectStore.setConf(…) Diffs (updated) - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fcaffc7 Diff: https://reviews.apache.org/r/31249/diff/ Testing --- Thanks, XIAOBING ZHOU
[jira] [Created] (HIVE-9757) LLAP: Reuse string dictionaries per column per stripe when processing row groups
Prasanth Jayachandran created HIVE-9757: --- Summary: LLAP: Reuse string dictionaries per column per stripe when processing row groups Key: HIVE-9757 URL: https://issues.apache.org/jira/browse/HIVE-9757 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Fix For: llap Dictionary streams are cached by low level cache. When creating column vectors from streams we create dictionary for every row group that we process. We should add per query cache to cache the deserialized representation of dictionary per stripe per column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9756) LLAP: use log4j 2 for llap
Gunther Hagleitner created HIVE-9756: Summary: LLAP: use log4j 2 for llap Key: HIVE-9756 URL: https://issues.apache.org/jira/browse/HIVE-9756 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gopal V For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get throughput friendly logging. http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9751) LLAP: Fix issue with reading last row group of string column
Prasanth Jayachandran created HIVE-9751: --- Summary: LLAP: Fix issue with reading last row group of string column Key: HIVE-9751 URL: https://issues.apache.org/jira/browse/HIVE-9751 Project: Hive Issue Type: Sub-task Affects Versions: llap Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Fix For: llap There is an issue with reading last row group of string column. This is mostly likely related to improper seeking into one of the streams. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9750) LLAP: avoid log locks in operators
Gunther Hagleitner created HIVE-9750: Summary: LLAP: avoid log locks in operators Key: HIVE-9750 URL: https://issues.apache.org/jira/browse/HIVE-9750 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Basically wrap all LOG.xx calls in isLogXXXEnabled to avoid unnecessary locks on these calls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9762) LLAP: hoop up decider + dag utils
Gunther Hagleitner created HIVE-9762: Summary: LLAP: hoop up decider + dag utils Key: HIVE-9762 URL: https://issues.apache.org/jira/browse/HIVE-9762 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9761) LLAP: Misc fixes to launch scripts, startup error handling
Siddharth Seth created HIVE-9761: Summary: LLAP: Misc fixes to launch scripts, startup error handling Key: HIVE-9761 URL: https://issues.apache.org/jira/browse/HIVE-9761 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9760) Killed job is not displayed correctly by failure hook
Xiaobing Zhou created HIVE-9760: --- Summary: Killed job is not displayed correctly by failure hook Key: HIVE-9760 URL: https://issues.apache.org/jira/browse/HIVE-9760 Project: Hive Issue Type: Bug Affects Versions: 1.0.0 Environment: CentOS Reporter: Xiaobing Zhou Assignee: Xiaobing Zhou PROBLEM: After killing the job throug CLI by crtl+c, we are seeing the following exception. Also the job is shown as running with timer still counting on the UI Interrupting... Be patient, this might take some time. Press Ctrl+C again to kill JVM Map 1: 20/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 22/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 25/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 26/40 Reducer 2: 0/3 Reducer 3: 0/1 Exiting the JVM Trying to shutdown DAG Shutting down tez session. Status: Killed Job received Kill while in RUNNING state. Vertex killed, vertexName=Reducer 3, vertexId=vertex_1421602925091_3868_2_00, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Map 1, vertexId=vertex_1421602925091_3868_2_02, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Reducer 2, vertexId=vertex_1421602925091_3868_2_01, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 DAG killed due to user-initiated kill. failedVertices:0 killedVertices:3 FAILED: Hive Internal Error: java.lang.IllegalStateException(Shutdown in progress) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at org.apache.hadoop.hive.ql.hooks.ATSHook.init(ATSHook.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1177) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1161) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1316) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1093) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) hive EL Info: 2015-01-21 16:20:51.483-ServerSession(1006268460)-EclipseLink, version: Eclipse Persistence Services - 2.5.1.v20130918-f2b9fc5 EL Info: connection: 2015-01-21 16:20:52.025-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger login successful EL Info: connection: 2015-01-21 16:20:52.082-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger logout successful -- This message was sent by Atlassian JIRA (v6.3.4#6332)
review HIVE-9741: Refactor MetaStoreDirectSql constructor by removing DB queries out of critical section
Hi team, Can you review the latest patch? https://issues.apache.org/jira/browse/HIVE-9741 https://reviews.apache.org/r/31249/
[jira] [Created] (HIVE-9763) Remove M* classes from RawStore interface
Alan Gates created HIVE-9763: Summary: Remove M* classes from RawStore interface Key: HIVE-9763 URL: https://issues.apache.org/jira/browse/HIVE-9763 Project: Hive Issue Type: Sub-task Components: Metastore Affects Versions: hbase-metastore-branch Reporter: Alan Gates Assignee: Alan Gates There are a number of methods that return M* classes (MDb, MRole), etc. These are an artifact of the ObjectStore interface and do not belong in RawStore. These methods should be refactored to instead return suitable thrift objects, especially since HiveMetaStore simply has to convert them before it can pass them across the wire. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9759) LLAP: Update launcher, scheduler to work with Tez changes
Siddharth Seth created HIVE-9759: Summary: LLAP: Update launcher, scheduler to work with Tez changes Key: HIVE-9759 URL: https://issues.apache.org/jira/browse/HIVE-9759 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31333: Killed job is not displayed correctly by failure hook
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31333/ --- Review request for hive. Repository: hive-git Description --- PROBLEM: After killing the job throug CLI by crtl+c, we are seeing the following exception. Also the job is shown as running with timer still counting on the UI Interrupting... Be patient, this might take some time. Press Ctrl+C again to kill JVM Map 1: 20/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 22/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 25/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 26/40 Reducer 2: 0/3 Reducer 3: 0/1 Exiting the JVM Trying to shutdown DAG Shutting down tez session. Status: Killed Job received Kill while in RUNNING state. Vertex killed, vertexName=Reducer 3, vertexId=vertex_1421602925091_3868_2_00, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Map 1, vertexId=vertex_1421602925091_3868_2_02, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Reducer 2, vertexId=vertex_1421602925091_3868_2_01, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 DAG killed due to user-initiated kill. failedVertices:0 killedVertices:3 FAILED: Hive Internal Error: java.lang.IllegalStateException(Shutdown in progress) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at org.apache.hadoop.hive.ql.hooks.ATSHook.init(ATSHook.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1177) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1161) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1316) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1093) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) hive EL Info: 2015-01-21 16:20:51.483-ServerSession(1006268460)-EclipseLink, version: Eclipse Persistence Services - 2.5.1.v20130918-f2b9fc5 EL Info: connection: 2015-01-21 16:20:52.025-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger login successful EL Info: connection: 2015-01-21 16:20:52.082-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger logout successful Diffs - ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java 513a2fa Diff: https://reviews.apache.org/r/31333/diff/ Testing --- Thanks, XIAOBING ZHOU
Re: Review Request 31333: Killed job is not display correctly by ATS hook failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31333/ --- (Updated Feb. 24, 2015, 1:19 a.m.) Review request for hive. Summary (updated) - Killed job is not display correctly by ATS hook failure Repository: hive-git Description --- PROBLEM: After killing the job throug CLI by crtl+c, we are seeing the following exception. Also the job is shown as running with timer still counting on the UI Interrupting... Be patient, this might take some time. Press Ctrl+C again to kill JVM Map 1: 20/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 22/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 25/40 Reducer 2: 0/3 Reducer 3: 0/1 Map 1: 26/40 Reducer 2: 0/3 Reducer 3: 0/1 Exiting the JVM Trying to shutdown DAG Shutting down tez session. Status: Killed Job received Kill while in RUNNING state. Vertex killed, vertexName=Reducer 3, vertexId=vertex_1421602925091_3868_2_00, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Map 1, vertexId=vertex_1421602925091_3868_2_02, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 Vertex killed, vertexName=Reducer 2, vertexId=vertex_1421602925091_3868_2_01, diagnostics=Vertex received Kill while in RUNNING state., Vertex killed due to user-initiated job kill. failedTasks:0 DAG killed due to user-initiated kill. failedVertices:0 killedVertices:3 FAILED: Hive Internal Error: java.lang.IllegalStateException(Shutdown in progress) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at org.apache.hadoop.hive.ql.hooks.ATSHook.init(ATSHook.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1177) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1161) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1316) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1093) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) hive EL Info: 2015-01-21 16:20:51.483-ServerSession(1006268460)-EclipseLink, version: Eclipse Persistence Services - 2.5.1.v20130918-f2b9fc5 EL Info: connection: 2015-01-21 16:20:52.025-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger login successful EL Info: connection: 2015-01-21 16:20:52.082-ServerSession(1006268460)-file:/usr/lib/hive/lib/xasecure-audit.jar_xa_server_url=jdbc:mysql://bkcttplpd036.verizon.com/xasecure_user=xalogger logout successful Diffs - ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java 513a2fa Diff: https://reviews.apache.org/r/31333/diff/ Testing --- Thanks, XIAOBING ZHOU
[jira] [Created] (HIVE-9765) LLAP: uber mode where applicable
Gunther Hagleitner created HIVE-9765: Summary: LLAP: uber mode where applicable Key: HIVE-9765 URL: https://issues.apache.org/jira/browse/HIVE-9765 Project: Hive Issue Type: Sub-task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner when task is small enough and it's the last reduce stage we can just run it in the AM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 31335: Correlating expression cannot contain unqualified column references
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31335/ --- Review request for hive. Bugs: HIVE-9734 https://issues.apache.org/jira/browse/HIVE-9734 Repository: hive-git Description --- The following valid ISO-SQL constructs will not parse Unsupported SubQuery Expression 'c1': Correlating expression cannot contain unqualified column references The statement will parse if the predicate is changed to where tjoin1.c1 in ( ) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select c1 from tjoin1) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select tjoin1.c1 from tjoin1) select tjoin1.rnum, tjoin1.c1, tjoin1.c2 from tjoin1 where c1 in ( select x.c1 from tjoin1 x) Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 1b6b33b ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/test/queries/clientnegative/subquery_unqual_corr_expr.q 99ff9ca ql/src/test/queries/clientpositive/subquery_unqual_corr_expr.q PRE-CREATION ql/src/test/results/clientnegative/subquery_unqual_corr_expr.q.out f69a538 ql/src/test/results/clientpositive/subquery_unqual_corr_expr.q.out PRE-CREATION Diff: https://reviews.apache.org/r/31335/diff/ Testing --- Thanks, Navis Ryu
[jira] [Created] (HIVE-9766) Add JavaConstantXXXObjectInspector
Daniel Dai created HIVE-9766: Summary: Add JavaConstantXXXObjectInspector Key: HIVE-9766 URL: https://issues.apache.org/jira/browse/HIVE-9766 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Reporter: Daniel Dai Assignee: Daniel Dai Attachments: HIVE-9766.1.patch Need JavaConstantXXXObjectInspector when implementing PIG-3294. There are two approaches: 1. Add those classes in Pig. However, most construct of the base class JavaXXXObjectInspector is default scope, need to change them to protected 2. Add those classes in Hive Approach 2 should be better since those classes might be useful to Hive as well. Attach a patch to provide them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31249: Refactor MetaStoreDirectSql constructor by removing DB queries out of critical section
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31249/ --- (Updated Feb. 24, 2015, 12:49 a.m.) Review request for hive. Summary (updated) - Refactor MetaStoreDirectSql constructor by removing DB queries out of critical section Repository: hive-git Description --- MetaStoreDirectSql constructor is querying DB to determine dbType, which leads to too many DB queries to make megastore slow as ObjectStore.setConf might be called frequently. Moreover, ObjectStore.setConf begins/ends with lock acquire/release, if the underlying DB hangs somehow, lock is never released and all hereafter incoming requests are blocked. Two points: 1. Using getProductName based JDBC driver to get dbType info. 2. Since metastore auto-creaton is disabled by default, it'd better bypass ensureDbInit() and runTestQuery() in order to avoid DB queries within critical section of setConf. Here’s stack trace: MetaStoreDirectSql.determineDbType(...) MetaStoreDirectSql.MetaStoreDirectSql(...) ObjectStore.initialize(...) ObjectStore.setConf(…) Diffs - metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 574141c metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java fcaffc7 Diff: https://reviews.apache.org/r/31249/diff/ Testing --- Thanks, XIAOBING ZHOU
can you review HIVE-9760: Killed job is not displayed correctly by failure hook
Hi team, Can anyone review this patch V1? Thanks! https://issues.apache.org/jira/browse/HIVE-9760 https://reviews.apache.org/r/31333/
[jira] [Created] (HIVE-9764) LLAP: Update tez dependency in branch
Siddharth Seth created HIVE-9764: Summary: LLAP: Update tez dependency in branch Key: HIVE-9764 URL: https://issues.apache.org/jira/browse/HIVE-9764 Project: Hive Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: llap -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9758) LLAP: fix wrapinputformat for client/server; fix server init to use config
Sergey Shelukhin created HIVE-9758: -- Summary: LLAP: fix wrapinputformat for client/server; fix server init to use config Key: HIVE-9758 URL: https://issues.apache.org/jira/browse/HIVE-9758 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Gunther Hagleitner 1) Null check right now prevents NPE when split generation tries to wrap input format and IO is not initialized. There should be a flag visible via interface somewhere that will allow us to check if we are inside the service instead. 2) Server startup disregards the io.enabled config setting. Not clear if it makes sense to run service w/io disabled, need to make it explicit (and remove the setting if not, cause it makes no sense except in q files). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
