damccorm commented on code in PR #37639:
URL: https://github.com/apache/beam/pull/37639#discussion_r3320542339
##########
.github/trigger_files/beam_PostCommit_Python.json:
##########
@@ -1,5 +1,10 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run.",
"pr": "38069",
+<<<<<<< dataflowClientTesting
Review Comment:
Looks like a bad merge
##########
sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:
##########
@@ -1252,95 +1265,3 @@ def
_verify_interpreter_version_is_supported(pipeline_options):
'using an unsupported version of Python interpreter, pass '
'--experiment use_unsupported_python_version pipeline option.' %
(_PYTHON_VERSIONS_SUPPORTED_BY_DATAFLOW, sys.version))
-
-
-# To enable a counter on the service, add it to this dictionary.
-# This is required for the legacy python dataflow runner, as portability
-# does not communicate to the service via python code, but instead via a
-# a runner harness (in C++ or Java).
-# TODO(https://github.com/apache/beam/issues/19433) : Remove this antipattern,
-# legacy dataflow python pipelines will break whenever a new cy_combiner type
-# is used.
-structured_counter_translations = {
Review Comment:
Was this always just dead code? Or am I missing a different corresponding
change to remove uses of this?
##########
sdks/python/setup.py:
##########
@@ -533,6 +533,7 @@ def get_portability_package_data():
'google-cloud-datastore>=2.0.0,<3',
'google-cloud-pubsub>=2.1.0,<3',
'google-cloud-storage>=2.18.2,<4',
+ 'google-cloud-dataflow-client>=0.13.0,<0.14.0',
Review Comment:
We can probably just remove from gpu_image_requirements.txt now
##########
sdks/python/apache_beam/runners/dataflow/internal/apiclient.py:
##########
@@ -466,18 +464,15 @@ def __init__(self, options, proto_pipeline):
if '{' in label:
label = ast.literal_eval(label)
for key, value in label.items():
- self.proto.labels.additionalProperties.append(
- dataflow.Job.LabelsValue.AdditionalProperty(
- key=key, value=value))
+ self.proto.labels.update({key: value})
Review Comment:
Confirming this was an intentional change - this is no longer nested within
additonal_properties?
##########
sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py:
##########
@@ -135,24 +132,19 @@ def _get_metric_key(self, metric):
# step name (only happens for unstructured-named metrics).
# 2. Unable to unpack [step] or [namespace]; which should only happen
# for unstructured names.
- step = _get_match(
- metric.name.context.additionalProperties,
- lambda x: x.key == STEP_LABEL).value
+ step = metric.name.context['step']
step = self._translate_step_name(step)
- except ValueError:
+ except (KeyError, ValueError):
pass
namespace = "dataflow/v1b3" # Try to extract namespace or add a default.
- try:
- namespace = _get_match(
- metric.name.context.additionalProperties,
- lambda x: x.key == 'namespace').value
- except ValueError:
- pass
+ carried_namespace = metric.name.context.get('namespace', None)
+ if carried_namespace:
+ namespace = carried_namespace
Review Comment:
```suggestion
namespace = metric.name.context.get('namespace', 'dataflow/v1b3')
```
I think we can simplify and get rid of `carried_namespace`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
