liferoad commented on code in PR #35692:
URL: https://github.com/apache/beam/pull/35692#discussion_r2242895341
##########
sdks/python/apache_beam/yaml/tests/create.yaml:
##########
@@ -81,3 +81,37 @@ pipelines:
- {sdk: MapReduce, year: 2004}
- {sdk: Flume}
- {sdk: MillWheel, year: 2008}
+
+ # Simple Create with explicit null value
+ - pipeline:
+ type: chain
+ transforms:
+ - type: Create
+ config:
+ elements:
+ - {sdk: MapReduce, year: 2004}
Review Comment:
@damccorm this is related to what we discussed. We already have
`ValidateWithSchema` for YAML, which can help validate the schema. But we do
not have a way to define the schema. For example, here, year could be NULL but
the inferred schema from Beam is Optional[Any], which cannot be accepted by
WriteToBigQuery since WriteToBigQuery does not allow any type. We should do
something like `ValidateWithSchema` that allows users to define the output
schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
