SEZ9 commented on issue #12268: URL: https://github.com/apache/seatunnel/issues/12268#issuecomment-6406939390
@goutamadwant thanks for the update and for posting D7–D9 to the dev@ thread. On the three decisions as now written: - **D7** — agreed: `level=column` adding no error codes, D4 still applying to the job, and unknown column data reported as `200` with `columnMetadata: NOT_CAPTURED` / per-table `UNKNOWN` is the right shape. One confirmation: I read "default response is unchanged" as no `columnMetadata` field at all unless `level=column` is requested — correct? - **D8** — fine as its own open row with the four transforms and the `UNKNOWN` fallback. Holding transform code until dev@ confirms is the right order. - **D9** — counting column data against the 8 MiB limit and still returning the table graph (`200`, `NOT_CAPTURED`) when the column cap is exceeded is good. From your table, 2,000 tables × 50 columns is exactly the 100,000-column cap and the whole-job column response lands at 7.8 MB, just under 8 MiB. Is there a job shape you'd expect to hit the 413 before the column cap? If so, a sentence in section 7 pointing to `table=` for large jobs would help; the 34–66 KB `table=` responses make that case well. Two asks on the Impact section: 1. Clarify what "opt-in" means for capture: is column data stored with the snapshot for every job (the 85 KB → 2.8 MB growth at 2,000 tables), or only when something is enabled? If it's always captured, please say so and note the ~42–46 ms at job start as the cost. 2. State whether the gzip figures are what the stored snapshot occupies or the raw size, so the limit is unambiguous. With those two in the body, D7–D9 look good to me. <!-- streview-comment:2039 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
