[jira] [Updated] (HIVE-22148) S3A delegation tokens are not added in the job config of the Compactor.
[ https://issues.apache.org/jira/browse/HIVE-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-22148: - Status: Patch Available (was: Open) [~anishek] , please take a look. Review board link: https://reviews.apache.org/r/71372/ > S3A delegation tokens are not added in the job config of the Compactor. > --- > > Key: HIVE-22148 > URL: https://issues.apache.org/jira/browse/HIVE-22148 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-22148.01.patch > > > Compactor job does not have the s3 delegation tokens, required to contact s3 > and causes the job to fail. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22148) S3A delegation tokens are not added in the job config of the Compactor.
[ https://issues.apache.org/jira/browse/HIVE-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-22148: - Attachment: HIVE-22148.01.patch > S3A delegation tokens are not added in the job config of the Compactor. > --- > > Key: HIVE-22148 > URL: https://issues.apache.org/jira/browse/HIVE-22148 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-22148.01.patch > > > Compactor job does not have the s3 delegation tokens, required to contact s3 > and causes the job to fail. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-22148) S3A delegation tokens are not added in the job config of the Compactor.
[ https://issues.apache.org/jira/browse/HIVE-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-22148: > S3A delegation tokens are not added in the job config of the Compactor. > --- > > Key: HIVE-22148 > URL: https://issues.apache.org/jira/browse/HIVE-22148 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > Compactor job does not have the s3 delegation tokens, required to contact s3 > and causes the job to fail. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21651) Move protobuf serde into hive-exec.
[ https://issues.apache.org/jira/browse/HIVE-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21651: - Status: Patch Available (was: Open) > Move protobuf serde into hive-exec. > --- > > Key: HIVE-21651 > URL: https://issues.apache.org/jira/browse/HIVE-21651 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-21651.01.patch, HIVE-21651.02.patch > > > The serde and input format is not accessible without doing an add jar or > modifying hive aux libs. Moving it to hive-exec will let us use the serde. > > Can't move the serde to hive/serde since it depends on ProtobufMessageWriter > which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21651) Move protobuf serde into hive-exec.
[ https://issues.apache.org/jira/browse/HIVE-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21651: - Attachment: HIVE-21651.02.patch > Move protobuf serde into hive-exec. > --- > > Key: HIVE-21651 > URL: https://issues.apache.org/jira/browse/HIVE-21651 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-21651.01.patch, HIVE-21651.02.patch > > > The serde and input format is not accessible without doing an add jar or > modifying hive aux libs. Moving it to hive-exec will let us use the serde. > > Can't move the serde to hive/serde since it depends on ProtobufMessageWriter > which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21651) Move protobuf serde into hive-exec.
[ https://issues.apache.org/jira/browse/HIVE-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16826597#comment-16826597 ] Harish Jaiprakash commented on HIVE-21651: -- [~anishek], please review. RB link: https://reviews.apache.org/r/70551/ > Move protobuf serde into hive-exec. > --- > > Key: HIVE-21651 > URL: https://issues.apache.org/jira/browse/HIVE-21651 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-21651.01.patch > > > The serde and input format is not accessible without doing an add jar or > modifying hive aux libs. Moving it to hive-exec will let us use the serde. > > Can't move the serde to hive/serde since it depends on ProtobufMessageWriter > which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21651) Move protobuf serde into hive-exec.
[ https://issues.apache.org/jira/browse/HIVE-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21651: - Attachment: HIVE-21651.01.patch > Move protobuf serde into hive-exec. > --- > > Key: HIVE-21651 > URL: https://issues.apache.org/jira/browse/HIVE-21651 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-21651.01.patch > > > The serde and input format is not accessible without doing an add jar or > modifying hive aux libs. Moving it to hive-exec will let us use the serde. > > Can't move the serde to hive/serde since it depends on ProtobufMessageWriter > which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21651) Move protobuf serde into hive-exec.
[ https://issues.apache.org/jira/browse/HIVE-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-21651: > Move protobuf serde into hive-exec. > --- > > Key: HIVE-21651 > URL: https://issues.apache.org/jira/browse/HIVE-21651 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > The serde and input format is not accessible without doing an add jar or > modifying hive aux libs. Moving it to hive-exec will let us use the serde. > > Can't move the serde to hive/serde since it depends on ProtobufMessageWriter > which is in hive-exec. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783184#comment-16783184 ] Harish Jaiprakash commented on HIVE-21362: -- Test failures not related, it has cleared once earlier. Only fixes were for codestyle errors after that. 'whitespace' errors are in generated file. Not sure how to exclude that. > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch, > HIVE-21362.03.patch, HIVE-21362.04.patch, HIVE-21362.05.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Attachment: HIVE-21362.05.patch > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch, > HIVE-21362.03.patch, HIVE-21362.04.patch, HIVE-21362.05.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Attachment: HIVE-21362.04.patch > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch, > HIVE-21362.03.patch, HIVE-21362.04.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Attachment: HIVE-21362.03.patch > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch, > HIVE-21362.03.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782953#comment-16782953 ] Harish Jaiprakash commented on HIVE-21362: -- Rename file name from TestProtos to SampleProtos since the generated file was being considered as a Test. > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch, > HIVE-21362.03.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782929#comment-16782929 ] Harish Jaiprakash commented on HIVE-21362: -- I'm getting 404 while trying to access checkstyle errors. The whitespace errors and asflicense missing are from the generated file. How do I exclude generated file from these checks? Or should I fix the generated file? > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782302#comment-16782302 ] Harish Jaiprakash commented on HIVE-21362: -- * Fixing code review comments. * Fixing checkstyle warnings. * Added another implementation to load protobuf from BytesWritable. > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Attachment: HIVE-21362.02.patch > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch, HIVE-21362.02.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21362) Add an input format and serde to read from protobuf files.
[
https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781212#comment-16781212
]
Harish Jaiprakash edited comment on HIVE-21362 at 3/1/19 2:18 AM:
--
Thanks, [~jdere].
bq. Is there a RB for this somewhere?
Created one forgot to link: https://reviews.apache.org/r/70075
bq. Does ProtoMessageSerDe.createStructObjectInspector() need to handle
repeated struct fields like createObjectInspector() does?
It does, createStructObjectInspector calls createObjectInspector which in turn
calls createStructObjectInspect for struct types.
bq. Also createObjectInspector() has a call to System.out.println(), please
remove that or convert to debug logging.
Will remove this, was debugging code.
bq. I don't quite get the proto.maptypes property - is this just some special
condition due to the data format you are trying to read, or is this the only
way to specify that a field is of Map type? If the latter, doesn't the
descriptor have a way to specify a map type?
Proto 2 compiler does not support map types. This is a way to configure
conversion of repeated struct(key, value) into map. That makes it
easier to process in hive too, no explode followed by a filter is required for
this and easier to extract several values from the map without joins.
was (Author: harishjp):
Thanks, [~jdere].
{noformat}
Is there a RB for this somewhere?
{noformat}
Created one forgot to link: https://reviews.apache.org/r/70075
{noformat}
Does ProtoMessageSerDe.createStructObjectInspector() need to handle repeated
struct fields like createObjectInspector() does?
{noformat}
It does, createStructObjectInspector calls createObjectInspector which in turn
calls createStructObjectInspect for struct types.
{noformat}
Also createObjectInspector() has a call to System.out.println(), please remove
that or convert to debug logging.
{noformat}
Will remove this, was debugging code.
{noformat}
I don't quite get the proto.maptypes property - is this just some special
condition due to the data format you are trying to read, or is this the only
way to specify that a field is of Map type? If the latter, doesn't the
descriptor have a way to specify a map type?
{noformat}
Proto 2 compiler does not support map types. This is a way to configure
conversion of repeated struct(key, value) into map. That makes it
easier to process in hive too, no explode followed by a filter is required for
this and easier to extract several values from the map without joins.
> Add an input format and serde to read from protobuf files.
> --
>
> Key: HIVE-21362
> URL: https://issues.apache.org/jira/browse/HIVE-21362
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-21362.01.patch
>
>
> Logs are being generated using the HiveProtoLoggingHook and tez
> ProtoHistoryLoggingService. These are sequence files written using
> ProtobufMessageWritable.
> Implement a SerDe and input format to be able to create tables using these
> files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21362) Add an input format and serde to read from protobuf files.
[
https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781212#comment-16781212
]
Harish Jaiprakash commented on HIVE-21362:
--
Thanks, [~jdere].
{noformat}
Is there a RB for this somewhere?
{noformat}
Created one forgot to link: https://reviews.apache.org/r/70075
{noformat}
Does ProtoMessageSerDe.createStructObjectInspector() need to handle repeated
struct fields like createObjectInspector() does?
{noformat}
It does, createStructObjectInspector calls createObjectInspector which in turn
calls createStructObjectInspect for struct types.
{noformat}
Also createObjectInspector() has a call to System.out.println(), please remove
that or convert to debug logging.
{noformat}
Will remove this, was debugging code.
{noformat}
I don't quite get the proto.maptypes property - is this just some special
condition due to the data format you are trying to read, or is this the only
way to specify that a field is of Map type? If the latter, doesn't the
descriptor have a way to specify a map type?
{noformat}
Proto 2 compiler does not support map types. This is a way to configure
conversion of repeated struct(key, value) into map. That makes it
easier to process in hive too, no explode followed by a filter is required for
this and easier to extract several values from the map without joins.
> Add an input format and serde to read from protobuf files.
> --
>
> Key: HIVE-21362
> URL: https://issues.apache.org/jira/browse/HIVE-21362
> Project: Hive
> Issue Type: Task
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Critical
> Attachments: HIVE-21362.01.patch
>
>
> Logs are being generated using the HiveProtoLoggingHook and tez
> ProtoHistoryLoggingService. These are sequence files written using
> ProtobufMessageWritable.
> Implement a SerDe and input format to be able to create tables using these
> files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Description: Logs are being generated using the HiveProtoLoggingHook and tez ProtoHistoryLoggingService. These are sequence files written using ProtobufMessageWritable. Implement a SerDe and input format to be able to create tables using these files. was: Logs are being generated using the HiveProtoLoggingHook and tez ProtoHistoryLoggingService. These are sequence file written using ProtobufMessageWritable. Implement a SerDe and input format to be able to create tables using these files. > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence files written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Status: Patch Available (was: Open) [~anishek], [~jdere], Please help in reviewing this patch. > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence file written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21362: - Attachment: HIVE-21362.01.patch > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21362.01.patch > > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence file written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21362) Add an input format and serde to read from protobuf files.
[ https://issues.apache.org/jira/browse/HIVE-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-21362: > Add an input format and serde to read from protobuf files. > -- > > Key: HIVE-21362 > URL: https://issues.apache.org/jira/browse/HIVE-21362 > Project: Hive > Issue Type: Task > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > > Logs are being generated using the HiveProtoLoggingHook and tez > ProtoHistoryLoggingService. These are sequence file written using > ProtobufMessageWritable. > Implement a SerDe and input format to be able to create tables using these > files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21244) NPE in Hive Proto Logger
[ https://issues.apache.org/jira/browse/HIVE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765711#comment-16765711 ] Harish Jaiprakash commented on HIVE-21244: -- The log message is incorrect, we are not closing a file, but logging into the new file. The old message was different, because it was logged before close. > NPE in Hive Proto Logger > > > Key: HIVE-21244 > URL: https://issues.apache.org/jira/browse/HIVE-21244 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-21244.1.patch > > > [https://github.com/apache/hive/blob/4ddc9de90b6de032d77709c9631ab787cef225d5/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L308] > can cause NPE. There is no uncaught exception handler for this thread. This > NPE can silently fail and drop the event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Resolution: Duplicate Status: Resolved (was: Patch Available) > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Status: Patch Available (was: Open) [~anishek], [~prasanth_j], Please take a look. Just 2 lines changed, hope tests are not required for this. > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-21250: - Attachment: HIVE-21250.01.patch > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21250) NPE in HiveProtoLoggingHook for eventPerFile mode.
[ https://issues.apache.org/jira/browse/HIVE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-21250: > NPE in HiveProtoLoggingHook for eventPerFile mode. > -- > > Key: HIVE-21250 > URL: https://issues.apache.org/jira/browse/HIVE-21250 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-21250.01.patch > > > When eventPerFile is enabled, writer is set to null after the first event, it > causes an NPE in the next path until handleTick comes back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651179#comment-16651179 ] Harish Jaiprakash commented on HIVE-20746: -- Updated patch with FindBugs fix. Test case failure not related. > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch, HIVE-20746.02.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20746: - Attachment: HIVE-20746.02.patch > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch, HIVE-20746.02.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649766#comment-16649766 ] Harish Jaiprakash commented on HIVE-20746: -- Review board link: https://reviews.apache.org/r/69021/ > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20746: - Status: Patch Available (was: Open) [~anishek], Please review. > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20746: - Component/s: HiveServer2 > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20746: - Attachment: HIVE-20746.01.patch > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20746.01.patch > > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20746) HiveProtoHookLogger does not close file at end of day.
[ https://issues.apache.org/jira/browse/HIVE-20746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-20746: > HiveProtoHookLogger does not close file at end of day. > -- > > Key: HIVE-20746 > URL: https://issues.apache.org/jira/browse/HIVE-20746 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > The file rotation happens with an event currently. If there are no queries > fired for a long time, then the file rotation does not happen and we do not > close the file. This causes the clients to poll for the file for an > indeterminate amount of time. If there are multiple hiveservers there is no > way to tell which file will get more data. Fix this to close at end of day. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19401) Correct Queue Name for LLAP should be updated in ATSHook of hive
[ https://issues.apache.org/jira/browse/HIVE-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587155#comment-16587155 ] Harish Jaiprakash commented on HIVE-19401: -- [~anishek], Please review, review board link: https://reviews.apache.org/r/68436/ > Correct Queue Name for LLAP should be updated in ATSHook of hive > > > Key: HIVE-19401 > URL: https://issues.apache.org/jira/browse/HIVE-19401 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: Harish Jaiprakash >Priority: Minor > Attachments: HIVE-19401.01.patch > > > The queue name provided as part of the ATSHook in hive currently by default > picks up the queue from *mapreduce.job.queuename* config parameter. > However for LLAP this is incorrect. Along with workload management which uses > *hive.server2.tez.interactive.queue* this problem becomes slightly more > complicated since we dont show the llap queues to users in yarn queues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19401) Correct Queue Name for LLAP should be updated in ATSHook of hive
[ https://issues.apache.org/jira/browse/HIVE-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-19401: Assignee: Harish Jaiprakash > Correct Queue Name for LLAP should be updated in ATSHook of hive > > > Key: HIVE-19401 > URL: https://issues.apache.org/jira/browse/HIVE-19401 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: Harish Jaiprakash >Priority: Minor > Attachments: HIVE-19401.01.patch > > > The queue name provided as part of the ATSHook in hive currently by default > picks up the queue from *mapreduce.job.queuename* config parameter. > However for LLAP this is incorrect. Along with workload management which uses > *hive.server2.tez.interactive.queue* this problem becomes slightly more > complicated since we dont show the llap queues to users in yarn queues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19401) Correct Queue Name for LLAP should be updated in ATSHook of hive
[ https://issues.apache.org/jira/browse/HIVE-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19401: - Attachment: HIVE-19401.01.patch > Correct Queue Name for LLAP should be updated in ATSHook of hive > > > Key: HIVE-19401 > URL: https://issues.apache.org/jira/browse/HIVE-19401 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: Harish Jaiprakash >Priority: Minor > Attachments: HIVE-19401.01.patch > > > The queue name provided as part of the ATSHook in hive currently by default > picks up the queue from *mapreduce.job.queuename* config parameter. > However for LLAP this is incorrect. Along with workload management which uses > *hive.server2.tez.interactive.queue* this problem becomes slightly more > complicated since we dont show the llap queues to users in yarn queues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19401) Correct Queue Name for LLAP should be updated in ATSHook of hive
[ https://issues.apache.org/jira/browse/HIVE-19401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19401: - Status: Patch Available (was: Open) > Correct Queue Name for LLAP should be updated in ATSHook of hive > > > Key: HIVE-19401 > URL: https://issues.apache.org/jira/browse/HIVE-19401 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: Harish Jaiprakash >Priority: Minor > Attachments: HIVE-19401.01.patch > > > The queue name provided as part of the ATSHook in hive currently by default > picks up the queue from *mapreduce.job.queuename* config parameter. > However for LLAP this is incorrect. Along with workload management which uses > *hive.server2.tez.interactive.queue* this problem becomes slightly more > complicated since we dont show the llap queues to users in yarn queues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17771) Implement commands to manage resource plan
[ https://issues.apache.org/jira/browse/HIVE-17771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583265#comment-16583265 ] Harish Jaiprakash commented on HIVE-17771: -- Ah sorry, most likely I had "remove unused local variables" settings on in Eclipse and hence this change. But the automatic removal code does not know if the method calls has any side effects so it leaves it around. > Implement commands to manage resource plan > -- > > Key: HIVE-17771 > URL: https://issues.apache.org/jira/browse/HIVE-17771 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17771.01.patch, HIVE-17771.02.patch, > HIVE-17771.03.patch, HIVE-17771.04.patch > > > Please see parent jira about llap workload management. > This jira is to implement create and show resource plan commands in hive to > configure resource plans for llap workload. The following are the proposed > commands implemented as part of the jira: > CREATE RESOURCE PLAN plan_name WITH QUERY_PARALLELISM parallelism; > SHOW RESOURCE PLAN plan_name; > SHOW RESOURCE PLANS; > ALTER RESOURCE PLAN plan_name SET QUERY_PARALLELISM = parallelism; > ALTER RESOURCE PLAN plan_name RENAME TO new_name; > ALTER RESOURCE PLAN plan_name ACTIVATE; > ALTER RESOURCE PLAN plan_name DISABLE; > ALTER RESOURCE PLAN plan_name ENABLE; > DROP RESOURCE PLAN; > It will be followed up with more jiras to manage pools, triggers and copy > resource plans. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20383) Invalid queue name and synchronisation issues in hive proto events hook.
[ https://issues.apache.org/jira/browse/HIVE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579439#comment-16579439 ] Harish Jaiprakash commented on HIVE-20383: -- Review board link: https://reviews.apache.org/r/68340/ > Invalid queue name and synchronisation issues in hive proto events hook. > > > Key: HIVE-20383 > URL: https://issues.apache.org/jira/browse/HIVE-20383 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20383.01.patch > > > Hive uses the same context object for the hooks for a given query, which > results in some synchronisation issues in HiveProtoLoggingHook: > * post event arrives before pre event processing starts, since the object is > shared, the event type for both events is post event, causes skipping of pre > event publishing. > * Causes concurrent modification exception while iterating perf object. > The queue name is extracted does not consider execution context, fix to > extract queue name from different config based on the execution mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20383) Invalid queue name and synchronisation issues in hive proto events hook.
[ https://issues.apache.org/jira/browse/HIVE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20383: - Status: Patch Available (was: Open) Fixed both the issues. [~anishek]/[~gopalv], please review. > Invalid queue name and synchronisation issues in hive proto events hook. > > > Key: HIVE-20383 > URL: https://issues.apache.org/jira/browse/HIVE-20383 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20383.01.patch > > > Hive uses the same context object for the hooks for a given query, which > results in some synchronisation issues in HiveProtoLoggingHook: > * post event arrives before pre event processing starts, since the object is > shared, the event type for both events is post event, causes skipping of pre > event publishing. > * Causes concurrent modification exception while iterating perf object. > The queue name is extracted does not consider execution context, fix to > extract queue name from different config based on the execution mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20383) Invalid queue name and synchronisation issues in hive proto events hook.
[ https://issues.apache.org/jira/browse/HIVE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20383: - Attachment: HIVE-20383.01.patch > Invalid queue name and synchronisation issues in hive proto events hook. > > > Key: HIVE-20383 > URL: https://issues.apache.org/jira/browse/HIVE-20383 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20383.01.patch > > > Hive uses the same context object for the hooks for a given query, which > results in some synchronisation issues in HiveProtoLoggingHook: > * post event arrives before pre event processing starts, since the object is > shared, the event type for both events is post event, causes skipping of pre > event publishing. > * Causes concurrent modification exception while iterating perf object. > The queue name is extracted does not consider execution context, fix to > extract queue name from different config based on the execution mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20383) Invalid queue name and synchronisation issues in hive proto events hook.
[ https://issues.apache.org/jira/browse/HIVE-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-20383: > Invalid queue name and synchronisation issues in hive proto events hook. > > > Key: HIVE-20383 > URL: https://issues.apache.org/jira/browse/HIVE-20383 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-20383.01.patch > > > Hive uses the same context object for the hooks for a given query, which > results in some synchronisation issues in HiveProtoLoggingHook: > * post event arrives before pre event processing starts, since the object is > shared, the event type for both events is post event, causes skipping of pre > event publishing. > * Causes concurrent modification exception while iterating perf object. > The queue name is extracted does not consider execution context, fix to > extract queue name from different config based on the execution mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20025) Clean-up of event files created by HiveProtoLoggingHook.
[ https://issues.apache.org/jira/browse/HIVE-20025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531098#comment-16531098 ] Harish Jaiprakash commented on HIVE-20025: -- +1, non binding. > Clean-up of event files created by HiveProtoLoggingHook. > > > Key: HIVE-20025 > URL: https://issues.apache.org/jira/browse/HIVE-20025 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: Hive, hooks, pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20025.01.patch, HIVE-20025.02.patch, > HIVE-20025.03.patch > > > Currently, HiveProtoLoggingHook write event data to hdfs. The number of files > can grow to very large numbers. > Since the files are created under a folder with Date being a part of the > path, hive should have a way to clean up data older than a certain configured > time / date. This can be a job that can run with as little frequency as just > once a day. > This time should be set to 1 week default. There should also be a sane upper > bound of # of files so that when a large cluster generates a lot of files > during a spike, we don't force the cluster fall over. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20011) Move away from append mode in proto logging hook
[ https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20011: - Status: Patch Available (was: Open) Fixing append mode in history logger, [~anishek], Please review. > Move away from append mode in proto logging hook > > > Key: HIVE-20011 > URL: https://issues.apache.org/jira/browse/HIVE-20011 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-20011.01.patch > > > Proto logging hook uses append mode with SequenceFile which does not work on > filesystems which does not support append. Move away to using sequence file > with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond > the length in NameNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20011) Move away from append mode in proto logging hook
[ https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524991#comment-16524991 ] Harish Jaiprakash edited comment on HIVE-20011 at 6/27/18 12:39 PM: Fixing append mode in history logger, [~anishek], Please review. https://reviews.apache.org/r/67754/ was (Author: harishjp): Fixing append mode in history logger, [~anishek], Please review. > Move away from append mode in proto logging hook > > > Key: HIVE-20011 > URL: https://issues.apache.org/jira/browse/HIVE-20011 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-20011.01.patch > > > Proto logging hook uses append mode with SequenceFile which does not work on > filesystems which does not support append. Move away to using sequence file > with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond > the length in NameNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20011) Move away from append mode in proto logging hook
[ https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-20011: - Attachment: HIVE-20011.01.patch > Move away from append mode in proto logging hook > > > Key: HIVE-20011 > URL: https://issues.apache.org/jira/browse/HIVE-20011 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > Attachments: HIVE-20011.01.patch > > > Proto logging hook uses append mode with SequenceFile which does not work on > filesystems which does not support append. Move away to using sequence file > with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond > the length in NameNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20011) Move away from append mode in proto logging hook
[ https://issues.apache.org/jira/browse/HIVE-20011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-20011: > Move away from append mode in proto logging hook > > > Key: HIVE-20011 > URL: https://issues.apache.org/jira/browse/HIVE-20011 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Critical > > Proto logging hook uses append mode with SequenceFile which does not work on > filesystems which does not support append. Move away to using sequence file > with flush. And use the reader with Long.MAX_VALUE to ensure it reads beyond > the length in NameNode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: HIVE-19921.02-branch-3.patch > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01-branch-3.patch, HIVE-19921.01.patch, > HIVE-19921.02-branch-3.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: HIVE-19921.01-branch-3.patch > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01-branch-3.patch, HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Status: Patch Available (was: Open) > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: (was: HIVE-19921.01.patch) > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: HIVE-19921.01.patch > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514782#comment-16514782 ] Harish Jaiprakash commented on HIVE-19921: -- [~anishek], Please review. https://reviews.apache.org/r/67619/ > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Description: The perf log should return duration instead of end time. The queue name should be llap queue for llap queries. was: The perf log return end time instead of duration. The queue name should be llap queue for llap queries. > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log should return duration instead of end time. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Component/s: HiveServer2 > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log return end time instead of duration. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19921: - Attachment: HIVE-19921.01.patch > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19921.01.patch > > > The perf log return end time instead of duration. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19921) Fix perf duration and queue name in HiveProtoLoggingHook
[ https://issues.apache.org/jira/browse/HIVE-19921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-19921: > Fix perf duration and queue name in HiveProtoLoggingHook > > > Key: HIVE-19921 > URL: https://issues.apache.org/jira/browse/HIVE-19921 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > The perf log return end time instead of duration. > The queue name should be llap queue for llap queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.08-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch,
> HIVE-19646.08-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.07-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: (was: HIVE-19646.07-branch-3.patch)
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.07-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch, HIVE-19646.07-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.06-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch,
> HIVE-19646.06-branch-3.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.05-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05-branch-3.patch, HIVE-19646.05.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.05.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch, HIVE-19646.05.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491858#comment-16491858
]
Harish Jaiprakash commented on HIVE-19646:
--
Resubmitting patch 03 as 04 for a rerun.
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.04.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: (was: HIVE-19646.03-branch-3.patch)
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch,
> HIVE-19646.04.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: (was: HIVE-19646.03-branch-3.patch)
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: (was: HIVE-19646.03.patch)
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490320#comment-16490320
]
Harish Jaiprakash commented on HIVE-19646:
--
The test failures are not related. The second message is about it rerunning the
tests on same patch. I do not know why its running on same patch. Will
re-upload branch patch.
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03-branch-3.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.03-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03-branch-3.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.03.patch
HIVE-19646.03-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch,
> HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Fix Version/s: 3.0.1
3.1.0
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Component/s: HiveServer2
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.03.patch
HIVE-19646.03-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch, HIVE-19646.03-branch-3.patch, HIVE-19646.03.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.01-branch-3.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01-branch-3.patch, HIVE-19646.01.patch,
> HIVE-19646.02.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16486973#comment-16486973
]
Harish Jaiprakash commented on HIVE-19646:
--
Creating review board entry: https://reviews.apache.org/r/67265/
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch, HIVE-19646.02.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.02.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch, HIVE-19646.02.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483741#comment-16483741
]
Harish Jaiprakash edited comment on HIVE-19646 at 5/22/18 10:09 AM:
Create patch with following changes:
* Always get filesystem object from the path, instead of storing it.
* Retry logic when logging fails.
* Moved the code copied from tez to the tez folders, this is to make sure we
can remove this code in a backwards compatible way, when hive moves to next
release tez version. SequenceFile writes the key and value class in the file
and prevents being read if the class package name is different.
[~anishek], Please review.
was (Author: harishjp):
Create patch with following changes:
* Always get filesystem object from the path, instead of storing it.
* Retry logic when logging fails.
* Moved the code copied from tez to the tez folders, this is to make sure we
can remove this code in a backwards compatible way, when hive moves to next
release tez version. SequenceFile writes the key and value class in the file
and prevents being read if the class package name is different.
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Status: Patch Available (was: Open)
Create patch with following changes:
* Always get filesystem object from the path, instead of storing it.
* Retry logic when logging fails.
* Moved the code copied from tez to the tez folders, this is to make sure we
can remove this code in a backwards compatible way, when hive moves to next
release tez version. SequenceFile writes the key and value class in the file
and prevents being read if the class package name is different.
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash reassigned HIVE-19646:
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19646) Filesystem closed error in HiveProtoLoggingHook
[
https://issues.apache.org/jira/browse/HIVE-19646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated HIVE-19646:
-
Attachment: HIVE-19646.01.patch
> Filesystem closed error in HiveProtoLoggingHook
> ---
>
> Key: HIVE-19646
> URL: https://issues.apache.org/jira/browse/HIVE-19646
> Project: Hive
> Issue Type: Bug
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
>Priority: Major
> Attachments: HIVE-19646.01.patch
>
>
> Exception in proto logging hook on secure cluster.
> {code}
> 2018-05-18T04:48:01,136 ERROR [Hive Hook Proto Log Writer 0]:
> hooks.HiveProtoLoggingHook (:()) - Error writing proto message for query
> hive_20180518043717_ca3ab4df-6cab-4920-aa44-2340ae246ad2, eventType:
> QUERY_SUBMITTED:
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:472)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1652)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
> ~[hadoop-hdfs-client-3.0.0.3.0.0.0-1298.jar:?]
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
> ~[hadoop-common-3.0.0.3.0.0.0-1298.jar:?]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getPathForDate(DatePartitionedLogger.java:89)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.DatePartitionedLogger.getWriter(DatePartitionedLogger.java:73)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.writeEvent(HiveProtoLoggingHook.java:283)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook$EventLogger.lambda$generateEvent$1(HiveProtoLoggingHook.java:274)
> ~[hive-exec-3.0.0.3.0.0.0-1298.jar:3.0.0.3.0.0.0-1298]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_161]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Attachment: HIVE-19288.02.patch > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch, HIVE-19288.02.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Status: Patch Available (was: Open) Implemented proto based logger. The logger, reader and writer was copied from tez, I wanted to reuse them but realized that code is in unrelease tez version and hence a copy right now. > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Attachment: HIVE-19288.01.patch > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-19288: > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353372#comment-16353372 ] Harish Jaiprakash commented on HIVE-18617: -- [~sershe]/[~prasanth_j] Please commit. The test case failures are unrelated. > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18617: - Status: Patch Available (was: Open) Fixed bug in Action parser for pool path. [~prasanth_j] Please review. > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18617: - Attachment: HIVE-18617.01.patch > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18617.01.patch > > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18617) Workload management Action parser does not generate the correct pool path.
[ https://issues.apache.org/jira/browse/HIVE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-18617: > Workload management Action parser does not generate the correct pool path. > -- > > Key: HIVE-18617 > URL: https://issues.apache.org/jira/browse/HIVE-18617 > Project: Hive > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > For expression: "MOVE TO plan_2.pool", generated path is "plan_2..", but > expected plan_2.pool -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332204#comment-16332204 ] Harish Jaiprakash commented on HIVE-18231: -- Fixed asflicense and checkstyle errors from previous comment. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch, > HIVE-18231.03.patch, HIVE-18231.04.patch, HIVE-18231.05.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.05.patch > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch, > HIVE-18231.03.patch, HIVE-18231.04.patch, HIVE-18231.05.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.04.patch > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch, > HIVE-18231.03.patch, HIVE-18231.04.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332193#comment-16332193 ] Harish Jaiprakash commented on HIVE-18231: -- * Adding parsing logic for action expressions. * Updating query parallelism validation. * Addressing review comments. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch, > HIVE-18231.03.patch, HIVE-18231.04.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.03.patch * Changed expression factory to use parser in ResourcePlanParser * Fixed error reporting and skipping token issues in ResourcePlanParser * Removed query parallelism validation. * Changed alloc fraction validation to have sum of child < 1 for pools to ensure parent pool has some capacity left. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch, > HIVE-18231.03.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.02.patch * Removing validation of query parallelism. * Changing alloc fraction validation so that, except for root level, sum(child alloc fraction) < 1. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-18231.01.patch, HIVE-18231.02.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18273) add LLAP-level counters for WM
[ https://issues.apache.org/jira/browse/HIVE-18273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326018#comment-16326018 ] Harish Jaiprakash commented on HIVE-18273: -- +1 for the code. > add LLAP-level counters for WM > -- > > Key: HIVE-18273 > URL: https://issues.apache.org/jira/browse/HIVE-18273 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18273.01.patch, HIVE-18273.02.patch, > HIVE-18273.patch > > > On query fragment level (like IO counters) > time queued as guaranteed; > time running as guaranteed; > time running as speculative. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Status: Patch Available (was: Open) > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > Attachments: HIVE-18231.01.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18231) validate resource plan - part 2 - validate action and trigger expressions
[ https://issues.apache.org/jira/browse/HIVE-18231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-18231: - Attachment: HIVE-18231.01.patch Adding patch to ensure grammar is compatible with ExpressionFactory and modifying the test cases to test the invalid conditions. [~sershe] [~prasanth_j], Please review. > validate resource plan - part 2 - validate action and trigger expressions > - > > Key: HIVE-18231 > URL: https://issues.apache.org/jira/browse/HIVE-18231 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Harish Jaiprakash > Attachments: HIVE-18231.01.patch > > > The expression factories can probably be moved to metastore or common. Action > should be easy to verify; counter names in triggers cannot be verified, but > at least the expression form can be -- This message was sent by Atlassian JIRA (v6.4.14#64029)
