[
https://issues.apache.org/jira/browse/BEAM-8402?focusedWorklogId=343019&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-343019
]
ASF GitHub Bot logged work on BEAM-8402:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Nov/19 00:06
Start Date: 14/Nov/19 00:06
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #9811:
[BEAM-8402] Create a class hierarchy to represent environments
URL: https://github.com/apache/beam/pull/9811#discussion_r346068305
##########
File path: sdks/python/apache_beam/transforms/environments.py
##########
@@ -0,0 +1,396 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Environments concepts.
+
+For internal use only. No backwards compatibility guarantees."""
+
+from __future__ import absolute_import
+
+import json
+
+from google.protobuf import message
+
+from apache_beam.portability import common_urns
+from apache_beam.portability import python_urns
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api import endpoints_pb2
+from apache_beam.utils import proto_utils
+
+__all__ = ['Environment',
+ 'DockerEnvironment', 'ProcessEnvironment', 'ExternalEnvironment',
+ 'EmbeddedPythonEnvironment', 'EmbeddedPythonGrpcEnvironment',
+ 'SubprocessSDKEnvironment', 'RunnerAPIEnvironmentHolder']
+
+
+class Environment(object):
+ """Abstract base class for environments.
+
+ Represents a type and configuration of environment.
+ Each type of Environment should have a unique urn.
+
+ For internal use only. No backwards compatibility guarantees.
+ """
+
+ _known_urns = {}
+ _urn_to_env_cls = {}
+
+ def to_runner_api_parameter(self, context):
+ raise NotImplementedError
+
+ @classmethod
+ def register_urn(cls, urn, parameter_type, constructor=None):
+
+ def register(constructor):
+ if isinstance(constructor, type):
+ constructor.from_runner_api_parameter = register(
+ constructor.from_runner_api_parameter)
+ # register environment urn to environment class
+ cls._urn_to_env_cls[urn] = constructor
+ return constructor
+
+ else:
+ cls._known_urns[urn] = parameter_type, constructor
+ return staticmethod(constructor)
+
+ if constructor:
+ # Used as a statement.
+ register(constructor)
+ else:
+ # Used as a decorator.
+ return register
+
+ @classmethod
+ def get_env_cls_from_urn(cls, urn):
+ return cls._urn_to_env_cls[urn]
+
+ def to_runner_api(self, context):
+ urn, typed_param = self.to_runner_api_parameter(context)
+ return beam_runner_api_pb2.Environment(
+ urn=urn,
+ payload=typed_param.SerializeToString()
+ if isinstance(typed_param, message.Message)
+ else typed_param if (isinstance(typed_param, bytes) or
+ typed_param is None)
+ else typed_param.encode('utf-8')
+ )
+
+ @classmethod
+ def from_runner_api(cls, proto, context):
+ if proto is None or not proto.urn:
+ return None
+ parameter_type, constructor = cls._known_urns[proto.urn]
+
+ try:
+ return constructor(
+ proto_utils.parse_Bytes(proto.payload, parameter_type),
+ context)
+ except Exception:
+ if context.allow_proto_holders:
+ return RunnerAPIEnvironmentHolder(proto)
+ raise
+
+ @classmethod
+ def from_options(cls, options):
+ """Creates an Environment object from PipelineOptions.
+
+ Args:
+ options: The PipelineOptions object.
+ """
+ raise NotImplementedError
+
+
+@Environment.register_urn(common_urns.environments.DOCKER.urn,
+ beam_runner_api_pb2.DockerPayload)
+class DockerEnvironment(Environment):
+
+ def __init__(self, container_image=None):
+ from apache_beam.runners.portability.portable_runner import PortableRunner
Review comment:
Seems like this creates a new dependency between DataflowRunner and
PortableRunner. Dataflow runner uses DockerEnvironment but there was no need
for DataflowRunner to depend on the PortableRunner module before. Can we do
this change without introducing the new dependency ? For example, can we
refactor out default_docker_image() method to this module ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 343019)
Time Spent: 4h 10m (was: 4h)
> Create a class hierarchy to represent environments
> --------------------------------------------------
>
> Key: BEAM-8402
> URL: https://issues.apache.org/jira/browse/BEAM-8402
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Chad Dombrova
> Assignee: Chad Dombrova
> Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> As a first step towards making it possible to assign different environments
> to sections of a pipeline, we first need to expose environment classes to the
> pipeline API. Unlike PTransforms, PCollections, Coders, and Windowings,
> environments exists solely in the portability framework as protobuf objects.
> By creating a hierarchy of "native" classes that represent the various
> environment types -- external, docker, process, etc -- users will be able to
> instantiate these and assign them to parts of the pipeline. The assignment
> portion will be covered in a follow-up issue/PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
