[
https://issues.apache.org/jira/browse/FLINK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717324#comment-17717324
]
Prabhu Joseph commented on FLINK-26425:
---------------------------------------
This is very useful to Flink on YARN, as without this, the rolled log files
(jobmanager.log.n or taskmanager.log.n) won't be aggregated into the HDFS/S3
Log Aggregation Path, which leads to a local disk full.
[SPARK-15990|https://issues.apache.org/jira/browse/SPARK-15990] has added this
support.
{quote}Could you give some examples of the rolling fashion under the settings
of spark.yarn.rolledLog.excludePattern and spark.yarn.rolledLog.includePattern?
{quote}
Example include and exclude pattern will look like below. YARN will aggregate
all rotated jobmanager log files and won't touch the out file.
flink.yarn.rolledLog.includePattern=jobmanager.log.*
flink.yarn.rolledLog.excludePattern=jobmanager.out
I can come up with a patch for this. Could you assign this to me? Thanks.
> Yarn aggregate log files in a rolling fashion for flink
> -------------------------------------------------------
>
> Key: FLINK-26425
> URL: https://issues.apache.org/jira/browse/FLINK-26425
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.14.3
> Reporter: wang da wei
> Priority: Major
>
> For the spark on yarn, you can set the
> spark.yarn.rolledLog.includePattern、spark.yarn.rolledLog.excludePattern to
> filter the log files and those log files will be aggregated in a rolling
> fashion,Do you have similar Settings in Flink？
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
