[jira] [Created] (HIVE-22148) S3A delegation tokens are not added in the job config of the Compactor.
Harish Jaiprakash created HIVE-22148: Summary: S3A delegation tokens are not added in the job config of the Compactor. Key: HIVE-22148 URL: https://issues.apache.org/jira/browse/HIVE-22148 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Compactor job does not have the s3 delegation tokens, required to contact s3 and causes the job to fail. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (HIVE-21651) Move protobuf serde into hive-exec.
Harish Jaiprakash created HIVE-21651: Summary: Move protobuf serde into hive-exec. Key: HIVE-21651 URL: https://issues.apache.org/jira/browse/HIVE-21651 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The serde and input format is not accessible without doing an add jar or modifying hive aux libs. Moving it to hive-exec will let us use the serde. Can't move the serde to hive/serde since it depends on ProtobufMessageWriter which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21362) Add an input format and serde to read from protobuf files.
Harish Jaiprakash created HIVE-21362: Summary: Add an input format and serde to read from protobuf files. Key: HIVE-21362 URL: https://issues.apache.org/jira/browse/HIVE-21362 Project: Hive Issue Type: Task Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Logs are being generated using the HiveProtoLoggingHook and tez ProtoHistoryLoggingService. These are sequence file written using ProtobufMessageWritable. Implement a SerDe and input format to be able to create tables using these files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
Harish Jaiprakash created HIVE-21250: Summary: NPE in HiveProtoLoggingHook for eventPerFile mode. Key: HIVE-21250 URL: https://issues.apache.org/jira/browse/HIVE-21250 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Attachments: HIVE-21250.01.patch When eventPerFile is enabled, writer is set to null after the first event, it causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
Harish Jaiprakash created HIVE-20746: Summary: HiveProtoHookLogger does not close file at end of day. Key: HIVE-20746 URL: https://issues.apache.org/jira/browse/HIVE-20746 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The file rotation happens with an event currently. If there are no queries fired for a long time, then the file rotation does not happen and we do not close the file. This causes the clients to poll for the file for an indeterminate amount of time. If there are multiple hiveservers there is no way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20383) Invalid queue name and synchronisation issues in hive proto events hook.
Harish Jaiprakash created HIVE-20383: Summary: Invalid queue name and synchronisation issues in hive proto events hook. Key: HIVE-20383 URL: https://issues.apache.org/jira/browse/HIVE-20383 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Attachments: HIVE-20383.01.patch Hive uses the same context object for the hooks for a given query, which results in some synchronisation issues in HiveProtoLoggingHook: * post event arrives before pre event processing starts, since the object is shared, the event type for both events is post event, causes skipping of pre event publishing. * Causes concurrent modification exception while iterating perf object. The queue name is extracted does not consider execution context, fix to extract queue name from different config based on the execution mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20011) Move away from append mode in proto logging hook
Harish Jaiprakash created HIVE-20011: Summary: Move away from append mode in proto logging hook Key: HIVE-20011 URL: https://issues.apache.org/jira/browse/HIVE-20011 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Proto logging hook uses append mode with SequenceFile which does not work on filesystems which does not support append. Move away to using sequence file with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond the length in NameNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
Harish Jaiprakash created HIVE-19921: Summary: Fix perf duration and queue name in HiveProtoLoggingHook Key: HIVE-19921 URL: https://issues.apache.org/jira/browse/HIVE-19921 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The perf log return end time instead of duration. The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
Harish Jaiprakash created HIVE-19646:
Summary: Filesystem closed error in HiveProtoLoggingHook
Key: HIVE-19646
URL: https://issues.apache.org/jira/browse/HIVE-19646
Project: Hive
Issue Type: Bug
Reporter: Harish Jaiprakash
Assignee: Harish Jaiprakash
Attachments: HIVE-19646.01.patch
Exception in proto logging hook on secure cluster.
{code}
2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
QUERY_SUBMITTED:
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
at
org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
at
org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
at
org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
at
org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-19288) Implement protobuf logging hive hook.
Harish Jaiprakash created HIVE-19288: Summary: Implement protobuf logging hive hook. Key: HIVE-19288 URL: https://issues.apache.org/jira/browse/HIVE-19288 Project: Hive Issue Type: Improvement Components: Hive Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Implement a protobuf based logger which will log hive hook events into date partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18630) Resource plan validate/activate does not verify trigger expressions.
Harish Jaiprakash created HIVE-18630: Summary: Resource plan validate/activate does not verify trigger expressions. Key: HIVE-18630 URL: https://issues.apache.org/jira/browse/HIVE-18630 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash Resource plan trigger expressions are not validated in WM validate. They are stored as strings in the database, standalone-metastore does not have access to the expression parser. This results in validate not checking the counter names and pool names in the validate/activate command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
Harish Jaiprakash created HIVE-18617: Summary: Workload management Action parser does not generate the correct pool path. Key: HIVE-18617 URL: https://issues.apache.org/jira/browse/HIVE-18617 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18210) create resource plan allows duplicates
Harish Jaiprakash created HIVE-18210:
Summary: create resource plan allows duplicates
Key: HIVE-18210
URL: https://issues.apache.org/jira/browse/HIVE-18210
Project: Hive
Issue Type: Bug
Reporter: Harish Jaiprakash
Create resource plan allows duplicates. This was seen in a cluster:
{code}
+--+---++
| rp_name | status | query_parallelism |
+--+---++
| plan_2 | ACTIVE| 10 |
| plan_2 | DISABLED | NULL |
+--+---++
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-18179) Implement validate resource plan.
Harish Jaiprakash created HIVE-18179: Summary: Implement validate resource plan. Key: HIVE-18179 URL: https://issues.apache.org/jira/browse/HIVE-18179 Project: Hive Issue Type: Sub-task Reporter: Harish Jaiprakash For now mostly pool alloc fraction to add up to 1.0 and some null or invalid value checks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17954) Implement create, alter and drop pool API's.
Harish Jaiprakash created HIVE-17954: Summary: Implement create, alter and drop pool API's. Key: HIVE-17954 URL: https://issues.apache.org/jira/browse/HIVE-17954 Project: Hive Issue Type: Sub-task Reporter: Harish Jaiprakash Implement pool management commands: CREATE POOL `resource_plan`.`pool_path` WITH ALLOC_FRACTION `fraction` QUERY_PARALLELISM `parallelism` SCHEDULING_POLICY `policy`; ALTER POOL `resource_plan`.`pool_path` SET PATH = `new_path`, ALLOC_FRACTION = `fraction`, QUERY_PARALLELISM = `parallelism`, SCHEDULING_POLICY = `policy`; DROP POOL `resource_plan`.`pool_path`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17910) Hive does not build, missing files in classification sub-module.
Harish Jaiprakash created HIVE-17910: Summary: Hive does not build, missing files in classification sub-module. Key: HIVE-17910 URL: https://issues.apache.org/jira/browse/HIVE-17910 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash hive does not build anymore for me, InterfaceAudience.java and InterfaceStability.java have been removed in HIVE-17743's commit. The patch looks fine, most likely a missed git add of classification. Please fix. cc [~stakiar]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17884) Implement create, alter and drop workload management triggers.
Harish Jaiprakash created HIVE-17884: Summary: Implement create, alter and drop workload management triggers. Key: HIVE-17884 URL: https://issues.apache.org/jira/browse/HIVE-17884 Project: Hive Issue Type: Sub-task Reporter: Harish Jaiprakash Implement triggers for workload management: The commands to be implemented: CREATE TRIGGER `resourceplan_name`.`trigger_name` WHEN condition DO action; condition is a boolean expression: variable operator value types with 'AND' and 'OR' support. action is currently: KILL or MOVE TO pool; ALTER TRIGGER `plan_name`.`trigger_name` WHEN condition DO action; DROP TRIGGER `plan_name`.`trigger_name`; Also add WM_TRIGGERS to information schema. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17839) Cannot generate thrift definitions in standalone-metastore.
Harish Jaiprakash created HIVE-17839: Summary: Cannot generate thrift definitions in standalone-metastore. Key: HIVE-17839 URL: https://issues.apache.org/jira/browse/HIVE-17839 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash mvn clean install -Pthriftif -Dthrift.home=... does not regenerate the thrift sources. This is after the https://issues.apache.org/jira/browse/HIVE-17506 fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17771) Implement create and show resource plan.
Harish Jaiprakash created HIVE-17771: Summary: Implement create and show resource plan. Key: HIVE-17771 URL: https://issues.apache.org/jira/browse/HIVE-17771 Project: Hive Issue Type: Sub-task Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17566) Create schema required for workload management.
Harish Jaiprakash created HIVE-17566: Summary: Create schema required for workload management. Key: HIVE-17566 URL: https://issues.apache.org/jira/browse/HIVE-17566 Project: Hive Issue Type: Sub-task Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Schema + model changes required for workload management. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17019) Add support to download debugging information as an archive.
Harish Jaiprakash created HIVE-17019: Summary: Add support to download debugging information as an archive. Key: HIVE-17019 URL: https://issues.apache.org/jira/browse/HIVE-17019 Project: Hive Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash -- This message was sent by Atlassian JIRA (v6.4.14#64029)
