[
https://issues.apache.org/jira/browse/FLINK-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010884#comment-17010884
]
Andrey Zagrebin commented on FLINK-15519:
-----------------------------------------
The latest suggestion from a joint offline discussion:
* build BashJavaUtils as a separate jar, packed with the log4j binder and its
fixed configuration
* the fixed logging configuration should point the logs to the stdout and
output minimal info level log strings, possibly w/o any prefixes
* put into e.g. /bin directory in flink-dist (not into lib to exclude
BashJavaUtils and its logging setup from the Flink class path)
* form the class path for running BashJavaUtils from its jar and
lib/flink-dist*.jar which should not have any binders
* save stdout of BashJavaUtils run into an env var of the parent bash process
* the output has to be tailed to some reasonable number of lines like 1000 to
avoid env var max size overflow
* tail the last line for the result
* read the log lines from env var in the subsequently started TM process and
log them according to the user logging setup line by line
* if BashJavaUtils fails then use default JVM args to start JVM process and
log the failure
We could just disable logs in the run of BashJavaUtils and call again
BashJavaUtils in the beginning of main standalone TM process to log but we
would like to avoid duplicated calls as the computation can get diverged
easily, like some later changes to Flink config in between etc.
> Preserve logs from BashJavaUtils and make them part of TM logs
> --------------------------------------------------------------
>
> Key: FLINK-15519
> URL: https://issues.apache.org/jira/browse/FLINK-15519
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
> Affects Versions: 1.10.0
> Reporter: Andrey Zagrebin
> Priority: Blocker
> Fix For: 1.10.0
>
>
> In FLINK-13983 we introduced BashJavaUtils utility to call in taskmanager.sh
> before starting TM and calculate memory configuration for the JVM process of
> TM.
> Ideally, it would be nice to preserve BashJavaUtils logs and make them part
> of the TM logs. Currently, logging for BashJavaUtils is configured from the
> class path and can differ from TM logging. Moreover TM logging can rewrite
> BashJavaUtils even if we align their loggings (e.g.
> log4j.appender.file.append=false in default log4j.properties for Flink).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
