[jira] [Created] (HIVE-24864) Hive Plugin (exec hook) to log application timeline data to ATSv2
Prabhu Joseph created HIVE-24864: Summary: Hive Plugin (exec hook) to log application timeline data to ATSv2 Key: HIVE-24864 URL: https://issues.apache.org/jira/browse/HIVE-24864 Project: Hive Issue Type: New Feature Reporter: Prabhu Joseph Assignee: Prabhu Joseph Currently HiveServer2 has ATSHook (https://issues.apache.org/jira/browse/HIVE-7076) to log query + plan information to ATSV1.5. Similar hook is needed for ATSv2. ATS1.5 has lot of scalability issues which will be improved by ATSv2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-20214) Support Capacity scheduler user queue mapping in non-impersonation mode
Prabhu Joseph created HIVE-20214: Summary: Support Capacity scheduler user queue mapping in non-impersonation mode Key: HIVE-20214 URL: https://issues.apache.org/jira/browse/HIVE-20214 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 1.2.2 Reporter: Prabhu Joseph Under non-impersonation mode, all MR / Tez jobs run as the hive system user. The default scheduler queue mapping is one queue per user. We need to provide a fix similar to HIVE-8424 for customers using Capacity Scheduler to use queue mapping with Hive non impersonation mode to have different queues for each end user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18440) CombineHiveInputFormat fails with IllegalArgumentException when the path is empty
Prabhu Joseph created HIVE-18440:
Summary: CombineHiveInputFormat fails with
IllegalArgumentException when the path is empty
Key: HIVE-18440
URL: https://issues.apache.org/jira/browse/HIVE-18440
Project: Hive
Issue Type: Bug
Reporter: Prabhu Joseph
Hive query failed with IllegalArgumentException when the number of paths is
zero. We create the fixedThreadPool with same value as number of paths and it
should greater than zero.
{code}
2018-01-05 00:32:55,745 [ERROR] [Dispatcher thread {Central}]
|impl.VertexImpl|: Vertex Input: share_of_shelf_kpi_values_int_01_001
initializer failed, vertex=vertex_1512053476937_11160_1_01 [Map 14]
org.apache.tez.dag.app.dag.impl.AMUserCodeException:
java.lang.IllegalArgumentException
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:319)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1140)
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at
com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135)
at
com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
at java.util.concurrent.FutureTask.setException(FutureTask.java:251)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1307)
at
java.util.concurrent.ThreadPoolExecutor.(ThreadPoolExecutor.java:1195)
at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:519)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:299)
at
org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:266)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
{code}
2018-01-05 00:32:55,724 [INFO] [InputInitializer {Map 14} #0]
|io.CombineHiveInputFormat|: Total number of paths: 0, launching 0 threads to
check non-combinable ones.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup
Prabhu Joseph created HIVE-18353:
Summary: CompactorMR should call jobclient.close() to trigger
cleanup
Key: HIVE-18353
URL: https://issues.apache.org/jira/browse/HIVE-18353
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.1
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
HiveMetastore process is leaking TrustStore reloader threads when running
compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618
and MAPREDUCE-6621
{code}
"Truststore reloader thread" #2814 daemon prio=1 os_prio=0
tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-16906) Hive ATSHook should check for yarn.timeline-service.enabled before connecting to ATS
Prabhu Joseph created HIVE-16906: Summary: Hive ATSHook should check for yarn.timeline-service.enabled before connecting to ATS Key: HIVE-16906 URL: https://issues.apache.org/jira/browse/HIVE-16906 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 1.2.2 Reporter: Prabhu Joseph Hive ATShook has to check yarn.timeline-service.enabled (Indicate to clients whether timeline service is enabled or not. If enabled, clients will put entities and events to the timeline server.) before creating TimelineClient -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-16185) TEZ_CLIENT_HOME is set wrongly
Prabhu Joseph created HIVE-16185:
Summary: TEZ_CLIENT_HOME is set wrongly
Key: HIVE-16185
URL: https://issues.apache.org/jira/browse/HIVE-16185
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 2.1.0
Reporter: Prabhu Joseph
TEZ_CLIENT_HOME is hardcoded with user directory instead of ${USER} in test
env.sh
https://github.com/apache/hive/blob/branch-2.1/hcatalog/src/test/e2e/templeton/deployers/env.sh#L58
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-9113) Explain on query failed with NPE
[
https://issues.apache.org/jira/browse/HIVE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248235#comment-14248235
]
Prabhu Joseph commented on HIVE-9113:
-
Hi Chao,
The subquery inside IN clause does not have from clause. The correct query
is below,
{noformat}
select p.p_partkey, li.suppkey
from (select distinct partkey as p_partkey from lineitem) p join lineitem li on
p.p_partkey = li.partkey
where li.l_linenumber = 1 and
li.l_orderkey in (select l_orderkey from lineitem where l_linenumber =
li.l_linenumber);
{noformat}
Explain on query failed with NPE
Key: HIVE-9113
URL: https://issues.apache.org/jira/browse/HIVE-9113
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Chao
Run explain on the following query:
{noformat}
select p.p_partkey, li.l_suppkey
from (select distinct l_partkey as p_partkey from lineitem) p join lineitem
li on p.p_partkey = li.l_partkey
where li.l_linenumber = 1 and
li.l_orderkey in (select l_orderkey where l_linenumber = li.l_linenumber)
;
{noformat}
gave me NPE:
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.parse.QBSubQuery.validateAndRewriteAST(QBSubQuery.java:516)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2605)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8866)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9745)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9638)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10125)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:420)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:306)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1108)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1035)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:720)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:639)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:578)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{noformat}
Is this query invalid? If so, it should at least give some explanations, not
just a plain NPE message, and left user clueless.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4286) When I run `CREATE TABLE new_table LIKE avro_table`, the schema does not get mapped properly from an Avro schema to a Hive schema
[ https://issues.apache.org/jira/browse/HIVE-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249447#comment-14249447 ] Prabhu Joseph commented on HIVE-4286: - Hi Uri Laserson, Which hive version this issue reproduced ?. In Hive 0.14, altering table itself throws error, ERROR exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table When I run `CREATE TABLE new_table LIKE avro_table`, the schema does not get mapped properly from an Avro schema to a Hive schema - Key: HIVE-4286 URL: https://issues.apache.org/jira/browse/HIVE-4286 Project: Hive Issue Type: Bug Reporter: Uri Laserson I have an Avro-backed table called vcf_flat. I run the following commands from the Hive shell: CREATE TABLE vcf_flat_whitelisted LIKE vcf_flat; ALTER TABLE vcf_flat_whitelisted SET FILEFORMAT SEQUENCEFILE; No errors, and I can see the table when I run `SHOW TABLES;`: hive SHOW TABLES; OK cosmic dbsnp sample_07 sample_08 vcf_flat vcf_flat_text vcf_flat_whitelisted Time taken: 0.061 seconds However, this is the output for `DESCRIBE vcf_flat_whitelisted`: FAILED: SemanticException [Error 10001]: Table not found vcf_flat_whitelisted -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7122) Storage format for create like table
[ https://issues.apache.org/jira/browse/HIVE-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247887#comment-14247887 ] Prabhu Joseph commented on HIVE-7122: - Hi Vasanth, Hive documentation tells like can be used to create table without copying data, it's just a view. But this improvement changes that feature and this feature can be achieved using Stored as CTAS. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-CreateTableLike Thanks, Prabhu Joseph Storage format for create like table Key: HIVE-7122 URL: https://issues.apache.org/jira/browse/HIVE-7122 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Vasanth kumar RJ Assignee: Vasanth kumar RJ Attachments: HIVE-7122.patch Using create like table user can specify the table storage format. Example: create table table1 like table2 stored as ORC; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8863) Cannot drop table with uppercase name after compute statistics for columns
[ https://issues.apache.org/jira/browse/HIVE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217631#comment-14217631 ] Prabhu Joseph commented on HIVE-8863: - Hi Juan, The issue got reproduced in my machine with hive 0.12 version. I am using Derby for metastore in server mode. Earlier when i drop the table i recieved different exception. drop table Test; The error i got is Caused by: MetaException(message:java.lang.RuntimeException: commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction) After applying HIVE-4996 patch, receiving same exception you have reported. drop table Test; NestedThrowablesStackTrace: java.sql.BatchUpdateException: DELETE on table 'TBLS' caused a violation of foreign key constraint 'TAB_COL_STATS_FK1' for key (19). But with or without applying patch, drop table test; [ small 't'] works. There is some issue. Cannot drop table with uppercase name after compute statistics for columns Key: HIVE-8863 URL: https://issues.apache.org/jira/browse/HIVE-8863 Project: Hive Issue Type: Bug Components: Metastore Reporter: Juan Yu Create a table with uppercase name Test, run analyze table Test compute statistics for columns col1 After this, you cannot drop the table by drop table Test; Got error: NestedThrowablesStackTrace: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (hive2.TAB_COL_STATS, CONSTRAINT TAB_COL_STATS_FK FOREIGN KEY (TBL_ID) REFERENCES TBLS (TBL_ID)) workaround is to use lowercase table name drop table test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8863) Cannot drop table with uppercase name after compute statistics for columns
[ https://issues.apache.org/jira/browse/HIVE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219059#comment-14219059 ] Prabhu Joseph commented on HIVE-8863: - Chaoyu, HIVE-6335 similar issue has been raised earlier, please close this as well. Cannot drop table with uppercase name after compute statistics for columns Key: HIVE-8863 URL: https://issues.apache.org/jira/browse/HIVE-8863 Project: Hive Issue Type: Bug Components: Metastore Reporter: Juan Yu Assignee: Chaoyu Tang Attachments: HIVE-8863.patch Create a table with uppercase name Test, run analyze table Test compute statistics for columns col1 After this, you cannot drop the table by drop table Test; Got error: NestedThrowablesStackTrace: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (hive2.TAB_COL_STATS, CONSTRAINT TAB_COL_STATS_FK FOREIGN KEY (TBL_ID) REFERENCES TBLS (TBL_ID)) workaround is to use lowercase table name drop table test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8863) Cannot drop table with uppercase name after compute statistics for columns
[ https://issues.apache.org/jira/browse/HIVE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214651#comment-14214651 ] Prabhu Joseph commented on HIVE-8863: - Hi Juan, I tried to reproduce the issue but it's working fine in hive-0.9.0-bin version with local mode. Below are the logs. hive create table Test (col INT); OK Time taken: 0.575 seconds hive analyze table Test compute statistics; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator 14/11/17 18:53:38 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/crsadmin/crsadmin_20141117185353_1a052b7c-2f4e-405f-8231-55bf43d9411b.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-2.0.0_cdh4.3.1/share/hadoop/mapreduce1/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-2.0.0_cdh4.3.1/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/HIVE/hive-0.9.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 0; number of reducers: 0 2014-11-17 18:53:46,365 null map = 0%, reduce = 0% Ended Job = job_local274860965_0001 Execution completed successfully Mapred Local Task Succeeded . Convert the Join into MapJoin Table default.test stats: [num_partitions: 0, num_files: 0, num_rows: 0, total_size: 0, raw_data_size: 0] OK Time taken: 13.004 seconds hive drop table Test; OK Can you help me providing simple steps to reproduce the issue and How much data present in Test table , what hive version , configuration details hive-site.xml of your environment. Thanks, Prabhu Joseph Cannot drop table with uppercase name after compute statistics for columns Key: HIVE-8863 URL: https://issues.apache.org/jira/browse/HIVE-8863 Project: Hive Issue Type: Bug Components: Metastore Reporter: Juan Yu Create a table with uppercase name Test, run analyze table Test compute statistics for columns col1 After this, you cannot drop the table by drop table Test; Got error: NestedThrowablesStackTrace: java.sql.BatchUpdateException: Cannot delete or update a parent row: a foreign key constraint fails (hive2.TAB_COL_STATS, CONSTRAINT TAB_COL_STATS_FK FOREIGN KEY (TBL_ID) REFERENCES TBLS (TBL_ID)) workaround is to use lowercase table name drop table test; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
