[
https://issues.apache.org/jira/browse/BEAM-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17547497#comment-17547497
]
Kenneth Knowles commented on BEAM-6734:
---------------------------------------
This issue has been migrated to https://github.com/apache/beam/issues/19258
> dataflow_metrics_test hard to maintain fake data
> ------------------------------------------------
>
> Key: BEAM-6734
> URL: https://issues.apache.org/jira/browse/BEAM-6734
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution
> Reporter: Alex Amato
> Priority: P3
>
>
> We have no way to dump and update the metric fake data this test uses. We
> should consider writing a tool to do this properly. We can change the format
> as long as its still human readable.
>
> I tried and the following failed:
> (1) Using json.dumps on the job_metrics
> Fails for not being JSON serializable
> (2) Using the proto json_format.py tools to dump on the job_metrics
> Fails with exception: object has no attribute 'DESCRIPTOR'
> [https://github.com/googleapis/google-cloud-python/issues/3485]
> (2) Using the proto text_format.py tools to dump on the job_metrics
> Seemed to dump an empty file?
> (3) Using this code to turn any object into a simple JSON style dict and
> print the JSON. This fails because it will print out all the private
> variables and other things we don't want. We could use this approach and
> allowlist the relevant fields.
> {code:java}
> def _create_simple_obj(self, obj):
> # If its a dictionary or defined class type
> if hasattr(obj, '__dict__') or isinstance(obj, dict):
> items = obj.items() if isinstance(obj, dict) else obj.__dict__.items()
> simple_dict = dict()
> for key, value in items:
> simple_dict[key] = self._create_simple_obj(value)
> return simple_dict
> elif isinstance(obj, (tuple, list, set, frozenset)):
> simple_list = []
> for x in obj:
> simple_list.append(self._create_simple_obj(x))
> return simple_list
> else:
> return obj
> def _pretty_format_job_metrics(self, job_metrics):
> job_metrics = self._create_simple_obj(obj)
> return json.dumps(obj, indent=4, sort_keys=True)
> {code}
>
>
> Or we can just give up and not try to dump new ones, just add some basic
> tests to make sure it can convert the metrics format to MetricResults.
>
> Then add more extensive testing on the MetricResults themselves with the
> matchers we are adding.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
