goutamadwant commented on issue #12268: URL: https://github.com/apache/seatunnel/issues/12268#issuecomment-6402743128
@SEZ9 thanks. Updated the body: - Dev list: D7–D9 posted to the same thread: https://lists.apache.org/thread/xff62okc0ncnrz1n3tp0gxbyk8sz7rkw - D1: the row now points to D7–D9 for column lineage. - D7: `level=column` adds no error codes. D4 applies to the job as before; unknown column data is a `200` with `columnMetadata: NOT_CAPTURED` or per-table `UNKNOWN` (section 7, Limits). - D8: its own open row, listing the four transforms and the `UNKNOWN` fallback; no transform code before dev@ confirms it. - D9: column data counts against the 8 MiB limit (413, whole response). Over the column cap, the table graph is still returned (`200`, `NOT_CAPTURED`). Measured on a local prototype (source → `Filter` → sink, 50 columns per table): | Job size | Extra time at job start | Stored snapshot | Table-level response | Column response (whole job) | Column response (`table=`) | |---|---|---|---|---|---| | 10 tables × 20 columns | < 1 ms | 2 KB → 9 KB | 0.9 KB | 17 KB | 2.5 KB | | 100 tables | ~10 ms | 6 KB → 144 KB | 3 KB | 392 KB | 7 KB | | 1,000 tables | ~46 ms | 43 KB → 1.4 MB (0.33 MB gzip) | 30 KB | 3.9 MB | 34 KB | | 2,000 tables | ~42 ms | 85 KB → 2.8 MB (0.66 MB gzip) | 62 KB | 7.8 MB | 66 KB | D3 (eager snapshot after `JobMaster` init) is still the item Phase 2 needs from you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
