[
https://issues.apache.org/jira/browse/HIVE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401311#comment-15401311
]
Sushanth Sowmyan edited comment on HIVE-14394 at 7/31/16 7:18 PM:
------------------------------------------------------------------
The recent event that predicated the filing of this bug is reports that we're
filling up disks of the metastore box from recent hive installations, due to
the excessive amount of metrics logging we generate. At first, this is not a
hive problem, because the primary bit of extra logging is coming from
HadoopMetrics2Reporter, which is a new library we depend on to publish our
Codahale metrics to the Hadoop Metrics2 subsystem, and is easily fixed by
changing the loglevel for that class to DEBUG, rather than retaining it on
INFO. However, it becomes our problem for upgrade scenarios where the customer
now has a manual change they need to make for smooth operations of their
cluster because we upgraded a dependency.
Thus, we need to work with the maintainer ([~elserj]) of HadoopMetrics2Reporter
to get a new version of the library, with info-level logging reduced to
debug-level logging. To this end, I have filed a github pull request :
https://github.com/joshelser/dropwizard-hadoop-metrics2/pull/1 .
However, in the meanwhile, we also need to make a patch available (even if we
don't commit it to hive) to solve the problem those installations have. Thus,
I'm uploading a temporary patch to move the Metrics2 Reporter to the hive
project, with the fix, so that those that need it can use this patch. I will
update this patch with a newer patch which bumps up the version number of the
dependency with the log-level aspect changed once we have a newer library
version with that.
was (Author: sushanth):
The recent event that predicated the filing of this bug is reports that we're
filling up disks of the metastore box from recent hive installations, due to
the excessive amount of metrics logging we generate. At first, this is not a
hive problem, because the primary bit of extra logging is coming from
HadoopMetrics2Reporter, which is a new library we depend on to publish our
Codahale metrics to the Hadoop Metrics2 subsystem, and is easily fixed by
changing the loglevel for that class to DEBUG, rather than retaining it on
INFO. However, it becomes our problem for upgrade scenarios where the customer
now has a manual change they need to make for smooth operations of their
cluster because we upgraded a dependency.
Thus, we need to work with the maintainer ([~elserj] of HadoopMetrics2Reporter
to get a new version of the library, with info-level logging reduced to
debug-level logging. To this end, I have filed a github pull request :
https://github.com/joshelser/dropwizard-hadoop-metrics2/pull/1 .
However, in the meanwhile, we also need to make a patch available (even if we
don't commit it to hive) to solve the problem those installations have. Thus,
I'm uploading a temporary patch to move the Metrics2 Reporter to the hive
project, with the fix, so that those that need it can use this patch. I will
update this patch with a newer patch which bumps up the version number of the
dependency with the log-level aspect changed once we have a newer library
version with that.
> Reduce excessive INFO level logging
> -----------------------------------
>
> Key: HIVE-14394
> URL: https://issues.apache.org/jira/browse/HIVE-14394
> Project: Hive
> Issue Type: Bug
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
>
> We need to cull down on the number of logs we generate in HMS and HS2 that
> are not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
