[
https://issues.apache.org/jira/browse/BEAM-7084?focusedWorklogId=246800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246800
]
ASF GitHub Bot logged work on BEAM-7084:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/May/19 15:19
Start Date: 22/May/19 15:19
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #8639: [BEAM-7084]
Allow custom environments for external transforms
URL: https://github.com/apache/beam/pull/8639#discussion_r286547528
##########
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##########
@@ -95,31 +95,40 @@ def default_docker_image():
@staticmethod
def _create_environment(options):
portable_options = options.view_as(PortableOptions)
+ return PortableRunner._resolve_environment(
+ portable_options.environment_type,
+ portable_options.environment_config)
+
+ @staticmethod
+ def _resolve_environment(environment_type, environment_config):
environment_urn = common_urns.environments.DOCKER.urn
- if portable_options.environment_type == 'DOCKER':
+ if environment_type == 'DOCKER':
environment_urn = common_urns.environments.DOCKER.urn
- elif portable_options.environment_type == 'PROCESS':
+ elif environment_type == 'PROCESS':
environment_urn = common_urns.environments.PROCESS.urn
- elif portable_options.environment_type in ('EXTERNAL', 'LOOPBACK'):
+ elif environment_type in ('EXTERNAL', 'LOOPBACK'):
environment_urn = common_urns.environments.EXTERNAL.urn
- elif portable_options.environment_type:
- if portable_options.environment_type.startswith('beam:env:'):
- environment_urn = portable_options.environment_type
+ elif environment_type == 'EMBEDDED':
+ environment_urn = environment_type
+ elif environment_type:
+ # Allow embedded environment for testing
+ if environment_type.startswith('beam:env:') or \
+ environment_type == 'EMBEDDED':
Review comment:
It doesn't have to be a standard URN, but if we're spreading it across
multiple codebases we should probably qualify it because it has a specific
meaning. Maybe "beam:env:embedded_java_for_testing:v1" or something like that
then. It'd break in strange ways if I tried to use this with a python expansion
service, or on a non-Java runner. What you really want to say is "run this as
cheaply as possible."
(Also, I still think it's inconsistent to have a "EMBEDDED" mean a
java-runner-specific thing thing and "DOCKER" a python-image-spawning one and
"PROCESS" something else altogether.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 246800)
Time Spent: 2h 40m (was: 2.5h)
> Support custom environment types when expanding external transforms
> -------------------------------------------------------------------
>
> Key: BEAM-7084
> URL: https://issues.apache.org/jira/browse/BEAM-7084
> Project: Beam
> Issue Type: Bug
> Components: beam-model, sdk-java-core
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Labels: portability
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The {{ExpansionService}} registers a default Docker JAVA SDK environment.
> Users should be able to specify other types of environment, i.e.
> Process/External/Embedded.
> We can add an {{environment_type}} field to the {{ExpansionRequest}} Proto
> message.
> CC [~robertwb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
