[
https://issues.apache.org/jira/browse/FLINK-40109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116022#comment-18116022
]
Martijn Visser commented on FLINK-40109:
----------------------------------------
This still fails on master after b464b331dbd, in a new shape: the poll added by
this ticket runs
to its timeout instead of asserting false immediately.
nightly 2026-09-13 on master, Java 21, module misc, both testDetachedMode and
testDetachedModeSecureWithPreInstallKeytab:
https://github.com/apache/flink/actions/runs/34732018337
nightly 2026-09-11 on master, Hadoop 3.4.3, module misc:
https://github.com/apache/flink/actions/runs/34553043463
{code}
[ERROR] org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode --
Time elapsed: 86.50 s <<< ERROR!
java.util.concurrent.TimeoutException: Kerberos keytab login [Login successful
for user, using keytab file] not found in taskmanager.log for application
application_1789094225420_0003 within the timeout; inspect the uploaded
container logs.
at
org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:218)
at
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyResultContainsKerberosKeytab(YARNSessionFIFOSecuredITCase.java:204)
{code}
Two of the sixteen master nightlies in the eight days to 2026-09-16. The
failing read is
taskmanager.log in one run and jobmanager.log in the other.
Separately, release-2.3 still has the pre-fix code and fails on the original
assertion
(verifyResultContainsKerberosKeytab:201, "Expecting value to be true but was
false"):
https://github.com/apache/flink/actions/runs/35046431137/job/104646790138
so this and FLINK-40099 need backporting to release-2.3 as well.
> YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab check
> runs before the YARN application is killed
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40109
> URL: https://issues.apache.org/jira/browse/FLINK-40109
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
> Affects Versions: 2.4.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 2.4.0
>
>
> {{YARNSessionFIFOSecuredITCase.testDetachedMode}} fails intermittently on
> master since FLINK-40099, e.g. GHA run 29005398820 (job 86077406039, module
> {{misc}}):
> {noformat}
> org.apache.flink.util.FlinkException: Exhausted retry attempts.
> at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:175)
> at
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyKerberosKeytabInLogs(YARNSessionFIFOSecuredITCase.java:209)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:201)
> {noformat}
> Root cause
> FLINK-40099 moved the Kerberos keytab log verification into a callback that
> runs after the job reaches FINISHED but before the application is killed, and
> polls the logs with {{waitUntilCondition(condition, 500)}}. Two problems:
> The {{500}} {{int}} literal binds to the {{waitUntilCondition(condition, int
> retryAttempts)}} overload (500 attempts × 100ms, then {{"Exhausted retry
> attempts."}}) rather than the intended millisecond interval.
> More fundamentally, reading the JobManager log while the application is still
> alive is racy. The container {{jobmanager.log}} that is in scope does not
> reliably contain the keytab login line before teardown: the bootstrap output
> (which carries the login line) and the running-JobManager output land in
> different log files, and the pre-kill content is not durably readable. The
> keytab line is present once the application has been torn down and its logs
> flushed.
> The original FLINK-17662 concern (a short-lived TaskManager's startup login
> line being briefly unreadable ~200ms after teardown) is real, but the fix for
> it should be a bounded post-kill poll, not relocating the check to before the
> kill.
> Proposed fix
> Revert FLINK-40099's pre-kill callback and read the container logs after the
> application is killed (output flushed), replacing the original single read
> with a bounded {{CommonTestUtils.waitUtil(...)}} poll that fails with a
> message naming the log file and the missing strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
