[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=133756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133756 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 10/Aug/18 18:36 Start Date: 10/Aug/18 18:36 Worklog Time Spent: 10m Work Description: pabloem closed pull request #6123: [BEAM-4028] Step in cleanup of create_operation function URL: https://github.com/apache/beam/pull/6123 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/runners/worker/operations.py b/sdks/python/apache_beam/runners/worker/operations.py index ac2780a5d87..0488fe928d3 100644 --- a/sdks/python/apache_beam/runners/worker/operations.py +++ b/sdks/python/apache_beam/runners/worker/operations.py @@ -572,15 +572,14 @@ def process(self, o): self.output(o) -def create_operation(name_context, spec, counter_factory, step_name, - state_sampler, test_shuffle_source=None, +def create_operation(name_context, spec, counter_factory, step_name=None, + state_sampler=None, test_shuffle_source=None, test_shuffle_sink=None, is_streaming=False): """Create Operation object for given operation specification.""" + + # TODO(pabloem): Document arguments to this function call. if not isinstance(name_context, common.NameContext): -# TODO(BEAM-4028): Remove ad-hoc NameContext once all has been migrated. -name_context = common.DataflowNameContext(step_name=name_context, - user_name=step_name, - system_name=None) +name_context = common.NameContext(step_name=name_context) if isinstance(spec, operation_specs.WorkerRead): if isinstance(spec.source, iobase.SourceBundle): This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 133756) Time Spent: 5h 40m (was: 5.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=133755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-133755 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 10/Aug/18 18:35 Start Date: 10/Aug/18 18:35 Worklog Time Spent: 10m Work Description: charlesccychen commented on issue #6123: [BEAM-4028] Step in cleanup of create_operation function URL: https://github.com/apache/beam/pull/6123#issuecomment-412169242 Thanks! This LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 133755) Time Spent: 5.5h (was: 5h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=130549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-130549 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 02/Aug/18 18:11 Start Date: 02/Aug/18 18:11 Worklog Time Spent: 10m Work Description: pabloem opened a new pull request #6123: [BEAM-4028] Step in cleanup of create_operation function URL: https://github.com/apache/beam/pull/6123 I'm cleaning up `step_name` arguments in favor of `name_context` arguments, and removing dataflow-specific code. r: @charlesccychen Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 130549) Time Spent: 5h 20m (was: 5h 10m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=95822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95822 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 27/Apr/18 03:33 Start Date: 27/Apr/18 03:33 Worklog Time Spent: 10m Work Description: aaltay closed pull request #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py index fbc137cec4f..1e879a844a7 100644 --- a/sdks/python/apache_beam/runners/common.py +++ b/sdks/python/apache_beam/runners/common.py @@ -56,7 +56,7 @@ def __ne__(self, other): return not self == other def __repr__(self): -return 'NameContext(%s)' % self.__dict__() +return 'NameContext(%s)' % self.__dict__ def __hash__(self): return hash(self.step_name) @@ -101,7 +101,7 @@ def __hash__(self): return hash((self.step_name, self.user_name, self.system_name)) def __repr__(self): -return 'DataflowNameContext(%s)' % self.__dict__() +return 'DataflowNameContext(%s)' % self.__dict__ def logging_name(self): """Stackdriver logging relies on user-given step names (e.g. Foo/Bar).""" diff --git a/sdks/python/apache_beam/runners/worker/operation_specs.py b/sdks/python/apache_beam/runners/worker/operation_specs.py index bdafbeaf44a..d38bc7788fa 100644 --- a/sdks/python/apache_beam/runners/worker/operation_specs.py +++ b/sdks/python/apache_beam/runners/worker/operation_specs.py @@ -24,6 +24,7 @@ import collections from apache_beam import coders +from apache_beam.runners import common # This module is experimental. No backwards-compatibility guarantees. @@ -359,17 +360,57 @@ class MapTask(object): stage_name: The name of this map task execution stage. system_names: The system names of the step corresponding to each map task operation in the execution graph. -step_names: The names of the step corresponding to each map task operation. +step_names: The user-given names of the step corresponding to each map task + operation (e.g. Foo/Bar/ParDo). original_names: The internal name of a step in the original workflow graph. +name_contexts: A common.NameContext object containing name information + about a step. """ - def __init__( - self, operations, stage_name, system_names, step_names, original_names): + def __init__(self, operations, stage_name, + system_names=None, + step_names=None, + original_names=None, + name_contexts=None): self.operations = operations self.stage_name = stage_name -self.system_names = system_names -self.step_names = step_names -self.original_names = original_names +# TODO(BEAM-4028): Remove arguments other than name_contexts. +self.name_contexts = name_contexts or self._make_name_contexts( +original_names, step_names, system_names) + + @staticmethod + def _make_name_contexts(original_names, user_names, system_names): +# TODO(BEAM-4028): Remove method once map task relies on name contexts. +return [common.DataflowNameContext(step_name, user_name, system_name) +for step_name, user_name, system_name in zip(original_names, + user_names, + system_names)] + + @property + def system_names(self): +"""Returns a list containing the system names of steps. + +A System name is the name of a step in the optimized Dataflow graph. +""" +return [nc.system_name for nc in self.name_contexts] + + @property + def original_names(self): +"""Returns a list containing the original names of steps. + +An original name is the internal name of a step in the Dataflow graph +(e.g. 's2'). +""" +return [nc.step_name for nc in self.name_contexts] + + @property + def step_names(self): +"""Returns a list containing the user names of steps. + +In this context, a step name is the user-given name of a step in the +Dataflow graph (e.g. 's2'). +""" +return [nc.user_name for nc in self.name_contexts] def __str__(self): return '<%s %s steps=%s>' % (self.__class__.__name__, self.stage_name, diff --git a/sdks/python/apache_beam/runners/worker/operations.py b/sdks/python/apache_beam/runners/worker/operations.py index aac56402db8..b5a75a574f7 100644 --- a/sdks/python/apache_beam/runners/worker/operations.py +++
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=95284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95284 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 25/Apr/18 22:20 Start Date: 25/Apr/18 22:20 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384452304 I believe this is ready to merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95284) Time Spent: 5h (was: 4h 50m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=95274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-95274 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 25/Apr/18 21:53 Start Date: 25/Apr/18 21:53 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384446462 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 95274) Time Spent: 4h 50m (was: 4h 40m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=94805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94805 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 24/Apr/18 21:45 Start Date: 24/Apr/18 21:45 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384090292 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94805) Time Spent: 4h 40m (was: 4.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=94804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94804 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 24/Apr/18 21:44 Start Date: 24/Apr/18 21:44 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384088644 Run Seed Job. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94804) Time Spent: 4.5h (was: 4h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=94803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94803 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 24/Apr/18 21:40 Start Date: 24/Apr/18 21:40 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384089102 Ran seed job to update for changes from https://github.com/apache/beam/pull/5214 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94803) Time Spent: 4h 20m (was: 4h 10m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=94801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94801 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 24/Apr/18 21:39 Start Date: 24/Apr/18 21:39 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-384088644 Run Seed Job. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94801) Time Spent: 4h 10m (was: 4h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=94149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-94149 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 23/Apr/18 16:40 Start Date: 23/Apr/18 16:40 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-383641550 I understand. The goal of NameContext is to isolate naming concerns as much as possible, so that any runner (dataflow / portability) needs only to provide an instance of NameContext. Once this is pulled in to Dataflow, I expect DataflowNameContext won't be needed in the shared beam code (only in the worker code that translates from map task) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 94149) Time Spent: 4h (was: 3h 50m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=92937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92937 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 19/Apr/18 23:45 Start Date: 19/Apr/18 23:45 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-382915146 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 92937) Time Spent: 3h 50m (was: 3h 40m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=91777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91777 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 17/Apr/18 15:14 Start Date: 17/Apr/18 15:14 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-382030061 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91777) Time Spent: 3h 40m (was: 3.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=91471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91471 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 16/Apr/18 20:35 Start Date: 16/Apr/18 20:35 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135#issuecomment-381740269 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91471) Time Spent: 3.5h (was: 3h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=91447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-91447 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 16/Apr/18 19:41 Start Date: 16/Apr/18 19:41 Worklog Time Spent: 10m Work Description: pabloem opened a new pull request #5135: [BEAM-4028] Transitioning MapTask objects to NameContext URL: https://github.com/apache/beam/pull/5135 Transitioning the `operation_specs.MapTask` to rely fully on `NameContext` for step naming. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 91447) Time Spent: 3h 20m (was: 3h 10m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90871 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 13/Apr/18 16:29 Start Date: 13/Apr/18 16:29 Worklog Time Spent: 10m Work Description: aaltay closed pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py index 0bf5bac88b5..117ad3b6ef6 100644 --- a/sdks/python/apache_beam/runners/common.py +++ b/sdks/python/apache_beam/runners/common.py @@ -39,6 +39,76 @@ from apache_beam.utils.windowed_value import WindowedValue +class NameContext(object): + """Holds the name information for a step.""" + + def __init__(self, step_name): +"""Creates a new step NameContext. + +Args: + step_name: The name of the step. +""" +self.step_name = step_name + + def __eq__(self, other): +return self.step_name == other.step_name + + def __ne__(self, other): +return not self == other + + def __repr__(self): +return 'NameContext(%s)' % self.__dict__() + + def __hash__(self): +return hash(self.step_name) + + def metrics_name(self): +"""Returns the step name used for metrics reporting.""" +return self.step_name + + def logging_name(self): +"""Returns the step name used for logging.""" +return self.step_name + + +# TODO(BEAM-4028): Move DataflowNameContext to Dataflow internal code. +class DataflowNameContext(NameContext): + """Holds the name information for a step in Dataflow. + + This includes a step_name (e.g. s2), a user_name (e.g. Foo/Bar/ParDo(Fab)), + and a system_name (e.g. s2-shuffle-read34).""" + + def __init__(self, step_name, user_name, system_name): +"""Creates a new step NameContext. + +Args: + step_name: The internal name of the step (e.g. s2). + user_name: The full user-given name of the step (e.g. Foo/Bar/ParDo(Far)). + system_name: The step name in the optimized graph (e.g. s2-1). +""" +super(DataflowNameContext, self).__init__(step_name) +self.user_name = user_name +self.system_name = system_name + + def __eq__(self, other): +return (self.step_name == other.step_name and +self.user_name == other.user_name and +self.system_name == other.system_name) + + def __ne__(self, other): +return not self == other + + def __hash__(self): +return hash((self.step_name, self.user_name, self.system_name)) + + def __repr__(self): +return 'DataflowNameContext(%s)' % self.__dict__() + + def logging_name(self): +"""Stackdriver logging relies on user-given step names (e.g. Foo/Bar).""" +return self.user_name + + class LoggingContext(object): """For internal use only; no backwards-compatibility guarantees.""" diff --git a/sdks/python/apache_beam/runners/worker/operations.pxd b/sdks/python/apache_beam/runners/worker/operations.pxd index cb05c90d242..0aee3371ff0 100644 --- a/sdks/python/apache_beam/runners/worker/operations.pxd +++ b/sdks/python/apache_beam/runners/worker/operations.pxd @@ -39,6 +39,7 @@ cdef class ConsumerSet(Receiver): cdef class Operation(object): + cdef readonly name_context cdef readonly operation_name cdef readonly spec cdef object consumers diff --git a/sdks/python/apache_beam/runners/worker/operations.py b/sdks/python/apache_beam/runners/worker/operations.py index 0fa32e3c997..977d4bba095 100644 --- a/sdks/python/apache_beam/runners/worker/operations.py +++ b/sdks/python/apache_beam/runners/worker/operations.py @@ -104,34 +104,45 @@ class Operation(object): one or more receiver operations that will take that as input. """ - def __init__(self, operation_name, spec, counter_factory, state_sampler): + def __init__(self, name_context, spec, counter_factory, state_sampler): """Initializes a worker operation instance. Args: - operation_name: The system name assigned by the runner for this -operation. + name_context: A NameContext instance or string(deprecated), with the +name information for this operation. spec: A operation_specs.Worker* instance. counter_factory: The CounterFactory to use for our counters. state_sampler: The StateSampler for the current operation. """ -self.operation_name = operation_name +if isinstance(name_context, common.NameContext): + # TODO(BEAM-4028): Clean this up once it's completely migrated. + # We use the specific
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90870
]
ASF GitHub Bot logged work on BEAM-4028:
Author: ASF GitHub Bot
Created on: 13/Apr/18 16:28
Start Date: 13/Apr/18 16:28
Worklog Time Spent: 10m
Work Description: aaltay commented on a change in pull request #5043:
[BEAM-4028] Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#discussion_r181442300
##
File path: sdks/python/apache_beam/runners/worker/operations.py
##
@@ -104,34 +104,45 @@ class Operation(object):
one or more receiver operations that will take that as input.
"""
- def __init__(self, operation_name, spec, counter_factory, state_sampler):
+ def __init__(self, name_context, spec, counter_factory, state_sampler):
"""Initializes a worker operation instance.
Args:
- operation_name: The system name assigned by the runner for this
-operation.
+ name_context: A NameContext instance or string(deprecated), with the
+name information for this operation.
spec: A operation_specs.Worker* instance.
counter_factory: The CounterFactory to use for our counters.
state_sampler: The StateSampler for the current operation.
"""
-self.operation_name = operation_name
+if isinstance(name_context, common.NameContext):
+ # TODO(BEAM-4028): Clean this up once it's completely migrated.
+ # We use the specific operation name that is used for metrics and state
+ # sampling.
+ self.name_context = name_context
+else:
+ logging.info('Creating namecontext within operation')
Review comment:
Can we reduce this to a debug log? (Fine to do it in a future PR.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 90870)
Time Spent: 3h (was: 2h 50m)
> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user,
> system name...).
> Depending on the needs of different components (operations, logging, metrics,
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming
> information for a single step.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90649 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 13/Apr/18 00:21 Start Date: 13/Apr/18 00:21 Worklog Time Spent: 10m Work Description: charlesccychen commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-380983169 Thanks Pablo! This LGTM. R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90649) Time Spent: 2h 50m (was: 2h 40m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90644 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 13/Apr/18 00:17 Start Date: 13/Apr/18 00:17 Worklog Time Spent: 10m Work Description: pabloem commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r181257174 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -39,6 +39,75 @@ from apache_beam.utils.windowed_value import WindowedValue +class NameContext(object): + """Holds the name information for a step.""" + + def __init__(self, step_name): +"""Creates a new step NameContext. + +Args: + step_name: The name of the step. +""" +self.step_name = step_name + + def __eq__(self, other): +return self.step_name == other.step_name + + def __ne__(self, other): +return not self == other + + def __repr__(self): +return 'NameContext(%s)' % self.__dict__() + + def __hash__(self): +return hash(self.step_name) + + def metrics_name(self): +"""Returns the step name used for metrics reporting.""" +return self.step_name + + def logging_name(self): +"""Returns the step name used for logging.""" +return self.step_name + + +class DataflowNameContext(NameContext): Review comment: For now, yes. This is because the Dataflow-specific setup is done in `create_operation`. This will be followed by 1) internal change, and then 2)PR removing it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90644) Time Spent: 2.5h (was: 2h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90645 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 13/Apr/18 00:17 Start Date: 13/Apr/18 00:17 Worklog Time Spent: 10m Work Description: pabloem commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r181257183 ## File path: sdks/python/apache_beam/runners/worker/operations.py ## @@ -104,34 +104,45 @@ class Operation(object): one or more receiver operations that will take that as input. """ - def __init__(self, operation_name, spec, counter_factory, state_sampler): + def __init__(self, name_context, spec, counter_factory, state_sampler): """Initializes a worker operation instance. Args: - operation_name: The system name assigned by the runner for this -operation. + name_context: A NameContext instance or string(deprecated), with the +name information for this operation. spec: A operation_specs.Worker* instance. counter_factory: The CounterFactory to use for our counters. state_sampler: The StateSampler for the current operation. """ -self.operation_name = operation_name +if isinstance(name_context, common.NameContext): + #TODO(pabloem) - Clean this up once it's completely migrated. Review comment: Done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90645) Time Spent: 2h 40m (was: 2.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90643 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 13/Apr/18 00:17 Start Date: 13/Apr/18 00:17 Worklog Time Spent: 10m Work Description: pabloem commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r181257252 ## File path: sdks/python/apache_beam/runners/worker/operations.py ## @@ -682,14 +701,14 @@ def execute(self): # The order of the elements is important because the inputs use # list indexes as references. -step_names = ( -self._map_task.step_names or [None] * len(self._map_task.operations)) for ix, spec in enumerate(self._map_task.operations): # This is used for logging and assigning names to counters. - operation_name = self._map_task.system_names[ix] - step_name = step_names[ix] + name_context = common.DataflowNameContext( Review comment: No. The maptask will have the name contexts after internal change + PR, and this won't be necessary any longer. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90643) Time Spent: 2.5h (was: 2h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90259 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 12/Apr/18 01:40 Start Date: 12/Apr/18 01:40 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r180944557 ## File path: sdks/python/apache_beam/runners/worker/operations.py ## @@ -104,34 +104,45 @@ class Operation(object): one or more receiver operations that will take that as input. """ - def __init__(self, operation_name, spec, counter_factory, state_sampler): + def __init__(self, name_context, spec, counter_factory, state_sampler): """Initializes a worker operation instance. Args: - operation_name: The system name assigned by the runner for this -operation. + name_context: A NameContext instance or string(deprecated), with the +name information for this operation. spec: A operation_specs.Worker* instance. counter_factory: The CounterFactory to use for our counters. state_sampler: The StateSampler for the current operation. """ -self.operation_name = operation_name +if isinstance(name_context, common.NameContext): + #TODO(pabloem) - Clean this up once it's completely migrated. Review comment: Space before TODO. Can you also add a Jira for the remaining work? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90259) Time Spent: 2h 10m (was: 2h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90260 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 12/Apr/18 01:41 Start Date: 12/Apr/18 01:41 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r180944559 ## File path: sdks/python/apache_beam/runners/common.py ## @@ -39,6 +39,75 @@ from apache_beam.utils.windowed_value import WindowedValue +class NameContext(object): + """Holds the name information for a step.""" + + def __init__(self, step_name): +"""Creates a new step NameContext. + +Args: + step_name: The name of the step. +""" +self.step_name = step_name + + def __eq__(self, other): +return self.step_name == other.step_name + + def __ne__(self, other): +return not self == other + + def __repr__(self): +return 'NameContext(%s)' % self.__dict__() + + def __hash__(self): +return hash(self.step_name) + + def metrics_name(self): +"""Returns the step name used for metrics reporting.""" +return self.step_name + + def logging_name(self): +"""Returns the step name used for logging.""" +return self.step_name + + +class DataflowNameContext(NameContext): Review comment: Do we need this to be exposed in the Beam code? Can we have this only in the worker? If this needs to be done in two steps, can you add a TODO / Jira? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90260) Time Spent: 2h 20m (was: 2h 10m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90258 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 12/Apr/18 01:40 Start Date: 12/Apr/18 01:40 Worklog Time Spent: 10m Work Description: charlesccychen commented on a change in pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#discussion_r180944556 ## File path: sdks/python/apache_beam/runners/worker/operations.py ## @@ -682,14 +701,14 @@ def execute(self): # The order of the elements is important because the inputs use # list indexes as references. -step_names = ( -self._map_task.step_names or [None] * len(self._map_task.operations)) for ix, spec in enumerate(self._map_task.operations): # This is used for logging and assigning names to counters. - operation_name = self._map_task.system_names[ix] - step_name = step_names[ix] + name_context = common.DataflowNameContext( Review comment: Will we always need the Dataflow version here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 90258) Time Spent: 2h (was: 1h 50m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=89457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89457 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 10/Apr/18 15:58 Start Date: 10/Apr/18 15:58 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-380144985 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89457) Time Spent: 1h 40m (was: 1.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=89459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89459 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 10/Apr/18 15:58 Start Date: 10/Apr/18 15:58 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-380153786 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89459) Time Spent: 1h 50m (was: 1h 40m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=89451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-89451 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 10/Apr/18 15:34 Start Date: 10/Apr/18 15:34 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-380144985 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 89451) Time Spent: 1.5h (was: 1h 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88999
]
ASF GitHub Bot logged work on BEAM-4028:
Author: ASF GitHub Bot
Created on: 09/Apr/18 16:23
Start Date: 09/Apr/18 16:23
Worklog Time Spent: 10m
Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379810700
Adding that the failure occurs in hdfs_integration_test.sh, specifically in
`time docker-compose -p ${PROJECT_NAME} up --exit-code-from test \
--abort-on-container-exit`, and it's occurring everywhere, independently
of the changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 88999)
Time Spent: 1h 20m (was: 1h 10m)
> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user,
> system name...).
> Depending on the needs of different components (operations, logging, metrics,
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming
> information for a single step.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88650 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 07/Apr/18 00:03 Start Date: 07/Apr/18 00:03 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379415200 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88650) Time Spent: 1h 10m (was: 1h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88630 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 22:29 Start Date: 06/Apr/18 22:29 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379401511 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88630) Time Spent: 1h (was: 50m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88594 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 20:10 Start Date: 06/Apr/18 20:10 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379363801 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88594) Time Spent: 50m (was: 40m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88593=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88593 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 20:10 Start Date: 06/Apr/18 20:10 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88593) Time Spent: 40m (was: 0.5h) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Steps can have different names depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88556 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 18:37 Start Date: 06/Apr/18 18:37 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88556) Time Spent: 0.5h (was: 20m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Steps can have different steps depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88550 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 18:27 Start Date: 06/Apr/18 18:27 Worklog Time Spent: 10m Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759 Run Python PostCommits This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88550) Time Spent: 20m (was: 10m) > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Steps can have different steps depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class
[ https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88549 ] ASF GitHub Bot logged work on BEAM-4028: Author: ASF GitHub Bot Created on: 06/Apr/18 18:27 Start Date: 06/Apr/18 18:27 Worklog Time Spent: 10m Work Description: pabloem opened a new pull request #5043: [BEAM-4028] Adding NameContext to Python SDK. URL: https://github.com/apache/beam/pull/5043 Steps can have different names depending on the runner (stage, step, user, system name...). Depending on the needs of different components (operations, logging, metrics, statesampling) these step names are passed around without a specific order. This change starts by creating name context classes and passing them into operations. Next steps are: - Migrating operations away from using their operation names, and removing them from the object definition in `operations.pxd`, and clean all the operation creating logic. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 88549) Time Spent: 10m Remaining Estimate: 0h > Step / Operation naming should rely on a NameContext class > -- > > Key: BEAM-4028 > URL: https://issues.apache.org/jira/browse/BEAM-4028 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Steps can have different steps depending on the runner (stage, step, user, > system name...). > Depending on the needs of different components (operations, logging, metrics, > statesampling) these step names are passed around without a specific order. > Instead, SDK should rely on `NameContext` objects that carry all the naming > information for a single step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
