[
https://issues.apache.org/jira/browse/HBASE-18517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115375#comment-16115375
]
Vikas Vishwakarma commented on HBASE-18517:
-------------------------------------------
thanks [~stack] added a patch with 1k message width limit. At present we are
still investigating the issue seen in our setup, will update if we see
additional issues.
> limit max log message width in log4j
> ------------------------------------
>
> Key: HBASE-18517
> URL: https://issues.apache.org/jira/browse/HBASE-18517
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0, 1.5.0
> Reporter: Vikas Vishwakarma
> Assignee: Vikas Vishwakarma
> Fix For: 3.0.0, 1.5.0
>
> Attachments: HBASE-18374.master.001.patch,
> HBASE-18517.branch-1.001.patch
>
>
> We had two cases now in our prod / pilot setups which is leading to humongous
> log lines in RegionServer logs.
> In first case, one of the phoenix user had constructed a query with a really
> large list of Id filters (61 MB) that translated into HBase scan that was
> running slow which lead to responseTooSlow messages in the logs with the
> entire filter list being printed in the logs, example
> ipc.RpcServer - (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1501457864417,"responsesize":11,"method":"Scan","param":"region
> { type: REGION_NAME value: .....
> org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter\\022\351\\200\\036\\n(org.apache.phoenix.filter.SkipScanFilter
> ... <a really long filter list leading to single log lines of size 61MB> ...
> There was another case where a use case had created a table with really large
> key/region names. This was causing humongous log lines for flush and
> compaction on these regions filling up the RS logs
> These large logs usually cause issues with disk I/O load, loading the splunk
> servers, even machine perf degradations. With 61 MB log lines basic log
> processing commands like vim, scrolling the logs, wc -l , etc were getting
> stuck. High GC activity was also noted on this cluster although not 100% sure
> if it was related to above issue.
> We should consider limiting the message size in logs which can be easily done
> by adding a maximum width format modifier on the message conversion character
> in log4j.properties
> log4j.appender.console.layout.ConversionPattern=...: %m%n
> to
> log4j.appender.console.layout.ConversionPattern=...: %.10000m%n
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
