CryoThrust commented on issue #12268: URL: https://github.com/apache/seatunnel/issues/12268#issuecomment-5633623411
For the proposed REST contract, I'd recommend keeping the first version explicitly job-scoped and bounded, while leaving room for OpenLineage correlation later: * Use a stable response envelope with `job_id`, `execution_mode`, `generated_at`, `schema_version`, and a graph payload; do not expose raw connector configuration or sampled records. * Define node/edge identity independently of display names so retries, renamed aliases, and parallel subtasks do not create ambiguous lineage. * State whether the graph is a point-in-time snapshot or eventually consistent with the running job, and expose a capture timestamp / completeness flag. * Keep column-level lineage and cross-job dataset lineage out of the initial endpoint, but include optional opaque dataset identifiers that can later map to OpenLineage facets. * Bound graph size and return pagination or a clear `truncated` marker; a large streaming job must not turn one REST request into an unbounded response. * Add authorization/error semantics for unknown job, expired execution, and partially available DAG data before freezing the endpoint. This would make the REST API useful to operators now without coupling it to the UI response shape, while keeping the STIP-39 and OpenLineage proposals compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
