[
https://issues.apache.org/jira/browse/FLINK-16548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057067#comment-17057067
]
Yang Wang commented on FLINK-16548:
-----------------------------------
Hi [~chinan.hjc], i think it is a valid requirements.
Recently, Flink has bumped the log4j to log4j2. It is a good improvement. Now
some users could use log4j2 build-in appenders[1] or custom appender to collect
logs to external storage(e.g. HDFS, S3, etc.). However, it is hard to know some
very basic information to construct the log file name
\{component}_\{resourceId}.log.
What do we have now?
> resourceId from environment
Mesos: _FLINK_CONTAINER_ID
YARN: _FLINK_CONTAINER_ID
Native K8s: _FLINK_POD_NAME
Standalone: random generated id, do not have env
These environment could only work for TaskManager. For JobManager, Flink do not
have some specified environment to identify.
Maybe we need to set some unified environment to identify the role(JM/TM) and
id.
> [Requirement] Consistent id to identify jm/tm in different cluster modes
> ------------------------------------------------------------------------
>
> Key: FLINK-16548
> URL: https://issues.apache.org/jira/browse/FLINK-16548
> Project: Flink
> Issue Type: Wish
> Components: Runtime / Configuration
> Reporter: hejianchao
> Priority: Major
>
> Imagine that there is now s3-log-appender.jar, and logs can be written to s3
> with proper configuration.
> Then, in the same flink job, the exact same log4j configuration is used. How
> to make the key written to s3 for different jm / tm carry the id that
> uniquely identifies jm / tm? such as:
> s3://bucket/flink-logs/jobId/${jm-id}/logfile
> s3://bucket/flink-logs/jobId/${tm-1-id}/logfile
> s3://bucket/flink-logs/jobId/${tm-2-id}/logfile
> For the s3-log-appender here, it needs a way to sense the value of
> ${jm-id}/${tm-1-id}/${tm-2-id}, and it has the following claims :
> 1. This value can be used to locate the actual running jm, tm
> 2. The value is independent of cluster mode, whether standalone, yarn or
> kubernetes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
