SEZ9 commented on issue #12268:
URL: https://github.com/apache/seatunnel/issues/12268#issuecomment-5651010447
@goutamadwant thanks for the revision. The updated STIP now covers the
pieces that were blocking design review: snapshot-only reads with no DAG
reconstruction or connector loading, explicit active/history precedence,
deterministic 404/409/503 outcomes, the `GET /job-lineage/{jobId}` route with
`schemaVersion=1`, provisional structural/byte/concurrency/deadline limits, and
a topology-only field whitelist that excludes configuration, addresses, JAR
locations, SQL, traces, credentials, and exception details. That is a material
improvement, and it stays correctly separated from #12207 (no OpenLineage
receiver, column lineage, or new persistent lineage store).
@CryoThrust's suggestions line up well with where the proposal landed. A few
of them I'd like to see reflected explicitly in the STIP text if they are not
already:
- Node/edge identity defined independently of display names, so retries,
renamed aliases, and parallel subtasks cannot produce ambiguous lineage.
- The capture timestamp plus completeness flag you mentioned, stated clearly
as point-in-time snapshot semantics rather than eventually consistent with the
running job.
- The bounded-size behavior: whether the atomic size rejection maps to a
specific status, and whether a `truncated` marker or pagination is planned for
a later schema version.
- Optional opaque dataset identifiers as a forward-compatible hook for later
OpenLineage correlation, without adding any facets now.
To be clear on status: this remains a proposal, not acceptance or a green
light to implement. The remaining gate is the decision process recorded in
D1-D6. Concrete asks before anything else moves:
1. Publish a [DISCUSS] thread on the dev list and add the archive link here.
2. Record the decisions on identity, snapshot-only selection/history
association, route/error mapping, resource limits, and the access/overlap
boundary in the STIP.
3. Describe the contract tests that prove a cache miss cannot silently
trigger connector loading or unbounded reconstruction on a REST or scheduling
thread.
No implementation PR, cache warming, DAG reconstruction, connector loading,
or new persistent state should start until those are settled. Once the dev-list
link and recorded decisions are in, I'll do another pass on the contract.
<!-- streview-comment:1019 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
