ahmedabu98 opened a new pull request, #26100: URL: https://github.com/apache/beam/pull/26100
Development on external SchemaTransforms may lead to changes in how their configurations look (ie. adding and removing fields, changing the order of fields). In addition to that, currently most Java SchemaTransformProviders inherit from [TypedSchemaTransformProvider](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/TypedSchemaTransformProvider.java), which infers its configuration schema by using a configuration class and [AutoValueSchema](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java). While this approach is very convenient, the ordering of fields in the inferred schema is unfortunately not consistent. All of this is to say that the configuration schema of external transforms is prone to changes. When we use an external SchemaTransform in Python, we build a payload that includes the configuration fields. These are the same fields used to set up the external SchemaTransform. Currently, we only use the input kwargs to construct the payload, so we are blind to what the external configuration schema actually is. The changes in this PR make it so that we first fetch the external configuration schema then construct the payload in accordance to that schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
