reuvenlax opened a new pull request, #40437:
URL: https://github.com/apache/beam/pull/40437
When merging ValidatesRunner tests, we need to ensure that pipelines with
differing PipelineOptions (including and PIpelineOptions subclass) are not
merged together. This simple approach of just serializing the options
(MAPPER.valueToTree(options).get("options")), however that contains _every_
bound value. As a result, two options objects configured identically can
serialize differently depending purely on which getters happened to be called
on each — e.g. View.asList() and Reshuffle read updateCompatibilityVersion
during construction, and GcsFileSystem reads gcsPerformanceMetrics, so a
pipeline that uses them "has" those keys and one that doesn't, doesn't. In
these cases the getters simply return the default values, so the
PipelineOptions are logically the same.
There is a BoundValue.isDefault bit (private) that one might try to use to
solve this problem, by excluding default-values from the comparison. However
testing often roundtrips options through JSON, after which this isDefault bit
is lost.
This PR introduces
PipelineOptionsFactory.explicitlySetProperties(PipelineOptions) which returns
the names of properties that have been explicitly set. In order to maintain
things across roundtrips, we introduce a new field in the serialized JSON to
track any fields that have lazily-bound defaults. This is currently not enabled
in production pipelines - it's only enabled by TestPipeline.java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
