SEPURI-SAI-KRISHNA commented on PR #13329:
URL: https://github.com/apache/seatunnel/pull/13329#issuecomment-6999469466

   Thanks @nzw921rx. I read #13328, STIP-41 (#12457) and the full diff at 
`69f562a5ca`. The last commit already fixed the status cleanup case I had 
noted. +1 to Issue 4 from @SEZ9 (Hazelcast trust boundary), which I had noted 
too. One detail if a per-application secret is added: workers get the cluster 
name as a command-line argument (`WorkerContainerFactory`, 
`List.of(clusterName, masterAddress, ...)`), which other users on the 
NodeManager host can usually read with `ps`, so the secret should travel in the 
container credentials or a localized file instead. Points not covered there:
   
   **1. Scope vs. STIP-41.** The STIP lists Kerberos and pre-staged YARN 
distributions as non-goals for the initial YARN implementation; reviewers on 
#12457 asked for one self-contained archive per submission, with shared HDFS 
artifacts in a separate STIP. #13328 notes the Kerberos change. Named snapshots 
registered in place (`YarnDistributionSource.register`) are the pre-staged 
case. Could the #12457 reviewers confirm this scope? If not, the simple-auth 
path could land first, with Kerberos and snapshots as follow-ups.
   
   **2. Kerberos limits.** `YarnSecurityContext.masterTokens` gets tokens for 
the staging filesystem only, and the keytab stays on the submitter. The 
Kerberos section says the RM renews tokens up to the HDFS token lifetime, but 
not the effect: by default after 7 days 
(`dfs.namenode.delegation.token.max-lifetime`) the AM and workers lose HDFS 
access, so long streaming jobs stop writing HDFS checkpoints. Checkpoint 
storage on another nameservice gets no token. Please state both. Also, 
`YarnKerberosIT` checks launch-context tokens; no application runs on the 
secure cluster.
   
   **3. Smaller items**
   - The README says both suites read jobs from `src/test/resources/common`, 
which does not exist; YARN scenarios have their own `job.conf`.
   - No production callers: `YarnDistribution.stage(...)` (both overloads) and 
the static six-argument `YarnApplicationFileUploader.upload(...)` are unused 
(`submit` calls the instance `upload()`). `YarnContainerLaunchContextFactory`, 
`YarnDistribution.inspect(File)`, `archive(Path)` and the "older manifests" 
branch of `fromManifest` serve only tests.
   - The AM heartbeats every 500 ms for the whole run 
(`YarnResourceManagerDriver.java:65`). The #12460 review suggested 1 s.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to