Martijn Visser created FLINK-40109:
--------------------------------------
Summary: YARNSessionFIFOSecuredITCase.testDetachedMode is flaky:
Kerberos keytab check runs before the YARN application is killed
Key: FLINK-40109
URL: https://issues.apache.org/jira/browse/FLINK-40109
Project: Flink
Issue Type: Bug
Components: Deployment / YARN, Tests
Affects Versions: 2.4.0
Reporter: Martijn Visser
Assignee: Martijn Visser
Fix For: 2.4.0
{{YARNSessionFIFOSecuredITCase.testDetachedMode}} fails intermittently on
master since FLINK-40099, e.g. GHA run 29005398820 (job 86077406039, module
{{misc}}):
{noformat}
org.apache.flink.util.FlinkException: Exhausted retry attempts.
at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:175)
at
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyKerberosKeytabInLogs(YARNSessionFIFOSecuredITCase.java:209)
at
org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:201)
{noformat}
Root cause
FLINK-40099 moved the Kerberos keytab log verification into a callback that
runs after the job reaches FINISHED but before the application is killed, and
polls the logs with {{waitUntilCondition(condition, 500)}}. Two problems:
The {{500}} {{int}} literal binds to the {{waitUntilCondition(condition, int
retryAttempts)}} overload (500 attempts × 100ms, then {{"Exhausted retry
attempts."}}) rather than the intended millisecond interval.
More fundamentally, reading the JobManager log while the application is still
alive is racy. The container {{jobmanager.log}} that is in scope does not
reliably contain the keytab login line before teardown: the bootstrap output
(which carries the login line) and the running-JobManager output land in
different log files, and the pre-kill content is not durably readable. The
keytab line is present once the application has been torn down and its logs
flushed.
The original FLINK-17662 concern (a short-lived TaskManager's startup login
line being briefly unreadable ~200ms after teardown) is real, but the fix for
it should be a bounded post-kill poll, not relocating the check to before the
kill.
Proposed fix
Revert FLINK-40099's pre-kill callback and read the container logs after the
application is killed (output flushed), replacing the original single read with
a bounded {{CommonTestUtils.waitUtil(...)}} poll that fails with a message
naming the log file and the missing strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
