DanielLeens commented on PR #12209: URL: https://github.com/apache/seatunnel/pull/12209#issuecomment-6624353797
Thanks @SEZ9 for the re-check. I went through your list again on the current head `8434902b288fed7c6c06e315ab49583e7b2920e7`. Most of these points were fixed in commits that landed before `ca7867c583`, so they do not appear in the three-commit delta. Here is where each one lives on this head: - **F1 (job env redirecting the cluster token):** the guard is in the shared resolver, not in `JobMaster`. `LineageConfig.resolve` (`seatunnel-lineage/.../LineageConfig.java:131-150`) rejects a token in job env first (`:133`, `rejectJobAuthToken` at `:174`). When a token comes from the process env or cluster config, `receiverOption` (`:312-338`) takes `openlineage_url` and `openlineage_transport` only from those trusted layers. A job value that differs fails with "must match the environment or cluster receiver when an auth token is configured" (`:333-335`). So a submitter cannot send the cluster token to their own URL. `JobMaster.resolveLineageConfig` (`JobMaster.java:1166-1192`) goes through this path. `openlineage_enabled` can still be set by the job, but that only turns reporting on or off for the job's own run. It cannot change where the token goes. - **F5 (job-controlled timeout/retries):** `JobMaster.java:1180-1189` resolves the env/cluster values a second time without job options and throws if the job's `openlineage_timeout_ms` or `openlineage_retry_times` is above them. So these are bounded by the operator's limit. The job gets an error instead of a silent clamp. - **F2 (heartbeat metrics fan-out):** `getCurrentJobMetricsForLineage` (`JobMaster.java:1236-1287`) computes one deadline from `timeoutMs`, and each worker future uses `future.get(remaining, NANOSECONDS)`. If one worker is missing, slow or failing, that heartbeat's metrics are dropped with a WARN, and nothing blocks the lineage pool. In my reading, all-or-nothing is the safer choice: a partial sum would report counts that are too low as if they were complete. - **F4 (throttle timestamp):** the check before submit at `ZetaLineageReporter.java:119-123` only reads the timestamp. The CAS that advances it (`shouldReportHeartbeat`, `:157-171`) runs inside the queued task just before `emit` (`:139-141`). So a heartbeat that `JobMaster.java:1323` coalesces away never advances it. One gap remains: if `emit` itself fails after the CAS, that interval is skipped. Heartbeats are best-effort, so I see this as a possible small follow-up, not a blocker. - **F6 / F7 (docs):** `docs/en/engines/zeta/hybrid-cluster-deployment.md:129-150` no longer points to a Flink or known-limitations section. The cluster-level `openlineage` block is in `docs/en/engines/zeta/separated-cluster-deployment.md:185-207` and in the zh page at `:190`. - **F8 (token in `EngineConfig`):** `lineageOptions` is `@ToString.Exclude` (`EngineConfig.java:141`), so the Lombok `toString` does not print it. A typed holder would be a nice follow-up, but it is not needed for masking. If you have a concrete job config that still gets a token-bearing request to a URL the operator did not set, please share it and I will trace it. @88fantasy, about CI on this head: dispatch run [41568084994](https://github.com/88fantasy/seatunnel/actions/runs/41568084994) failed only in `unit-test (17, ubuntu-latest)`, on `Redis7Test.failedInitializationClosesConnections` (`Redis7Test.java:156` -> `:268`, a client-count wait). This PR does not touch `connector-redis`, and that test has not changed on `dev` since #12402. The connector IT jobs then show as cancelled after that failure. Re-running the failed and cancelled jobs of that run (`Re-run failed jobs`) should be enough; no code change is needed for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
