[
https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851674#comment-15851674
]
Jason Lowe commented on YARN-6125:
----------------------------------
I suppose we could do ellipses, but I'm still a bit confused. Are we both
truncating individual messages as they come in as well as the entire
diagnostics? I was under the impression we're not worrying about messages but
the diagnostics buffer as a whole. Whether they try to fill it with one big
message, a few messages, or one character at a time over a zillion calls, I
would expect it to look something like this, using the example 1 above as the
original diagnostics from the app:
{noformat}
...rter than the previour one
message3 this is even shorter
message4 the shortest one
{noformat}
Again I think it could be useful to convey to the user how much of the message
was truncated (i.e.: showing last N of X bytes) so they have an idea of how
much they're missing. But the first cut could just be ellipses at the start
and the last N bytes of the diagnostic messages where N is the configured
diagnostic limit.
> The application attempt's diagnostic message should have a maximum size
> -----------------------------------------------------------------------
>
> Key: YARN-6125
> URL: https://issues.apache.org/jira/browse/YARN-6125
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 2.7.0
> Reporter: Daniel Templeton
> Assignee: Andras Piros
> Priority: Critical
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6125.000.patch, YARN-6125.001.patch,
> YARN-6125.002.patch, YARN-6125.003.patch
>
>
> We've found through experience that the diagnostic message can grow
> unbounded. I've seen attempts that have diagnostic messages over 1MB. Since
> the message is stored in the state store, it's a bad idea to allow the
> message to grow unbounded. Instead, there should be a property that sets a
> maximum size on the message.
> I suspect that some of the ZK state store issues we've seen in the past were
> due to the size of the diagnostic messages and not to the size of the
> classpath, as is the current prevailing opinion.
> An open question is how best to prune the message once it grows too large.
> Should we
> # truncate the tail,
> # truncate the head,
> # truncate the middle,
> # add another property to make the behavior selectable, or
> # none of the above?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
