Alex Amato created BEAM-6734:
--------------------------------
Summary: dataflow_metrics_test hard to maintain fake data
Key: BEAM-6734
URL: https://issues.apache.org/jira/browse/BEAM-6734
Project: Beam
Issue Type: New Feature
Components: java-fn-execution
Reporter: Alex Amato
We have no way to dump and update the metric fake data this test uses. We
should consider writing a tool to do this properly. We can change the format as
long as its still human readable.
I tried and the following failed:
(1) Using json.dumps on the job_metrics
Fails for not being JSON serializable
(2) Using the proto json_format.py tools to dump on the job_metrics
Fails with exception: object has no attribute 'DESCRIPTOR'
[https://github.com/googleapis/google-cloud-python/issues/3485]
(2) Using the proto text_format.py tools to dump on the job_metrics
Seemed to dump an empty file?
(3) Using this code to turn any object into a simple JSON style dict and print
the JSON. This fails because it will print out all the private variables and
other things we don't want. We could use this approach and whitelist the
relevant fields.
{code:java}
def _create_simple_obj(self, obj):
# If its a dictionary or defined class type
if hasattr(obj, '__dict__') or isinstance(obj, dict):
items = obj.items() if isinstance(obj, dict) else obj.__dict__.items()
simple_dict = dict()
for key, value in items:
simple_dict[key] = self._create_simple_obj(value)
return simple_dict
elif isinstance(obj, (tuple, list, set, frozenset)):
simple_list = []
for x in obj:
simple_list.append(self._create_simple_obj(x))
return simple_list
else:
return obj
def _pretty_format_job_metrics(self, job_metrics):
job_metrics = self._create_simple_obj(obj)
return json.dumps(obj, indent=4, sort_keys=True)
{code}
Or we can just give up and not try to dump new ones, just add some basic tests
to make sure it can convert the metrics format to MetricResults.
Then add more extensive testing on the MetricResults themselves with the
matchers we are adding.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
