goutamadwant commented on issue #38513: URL: https://github.com/apache/beam/issues/38513#issuecomment-5308997827
@Abacn Can you confirm the direction of this task? I reviewed the current state and related PRs. #37631 was reverted by #38516, so Java currently does not emit `beam:coder:schema:v1`. The temporary test exclusions from #38497 were reverted as well. Could we keep this issue as the umbrella and split the implementation into linked tasks? 1. Model/conformance: add a shared `beam:coder:schema:v1` test vector and refresh generated SDK bindings for `SchemaCoderPayload`. 2. Python: parse the payload, support row-compatible encoding and decoding, advertise the URN, and preserve the original URN and complete payload during Runner API round trips. 3. Go: add the standard URN and generated payload type, support unmarshalling and row-compatible execution, advertise the capability, and preserve the payload when re-marshalling. 4. TypeScript: register a schema coder wrapper around `RowCoder`, preserve the original payload, and add registry and encoding tests. 5. Java: restore the SchemaCoder translator from #37631 only after the non-Java consumers and cross-SDK tests are ready. For non-origin SDKs, the `to_row_fn`, `from_row_fn`, and SDK-specific additional information should remain opaque. They should not attempt to deserialize Java function payloads. Simply converting the coder to a normal RowCoder would also be lossy because a subsequent Runner API serialization would drop the original URN and conversion metadata. I suggest treating consumption and lossless round-trip support as the first milestone. Native emission from Python, Go, and TypeScript can be separate follow-up work because that requires defining SDK-specific conversion function URNs and metadata. If this matches the intended scope, I would be happy to start with the shared conformance case and the Python task, since Python ULR exposed the original failure. let me know. thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
