[
https://issues.apache.org/jira/browse/BEAM-7084?focusedWorklogId=246766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246766
]
ASF GitHub Bot logged work on BEAM-7084:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/May/19 14:41
Start Date: 22/May/19 14:41
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #8639: [BEAM-7084]
Allow custom environments for external transforms
URL: https://github.com/apache/beam/pull/8639#discussion_r286527086
##########
File path: sdks/python/apache_beam/runners/portability/portable_runner.py
##########
@@ -95,31 +95,40 @@ def default_docker_image():
@staticmethod
def _create_environment(options):
portable_options = options.view_as(PortableOptions)
+ return PortableRunner._resolve_environment(
+ portable_options.environment_type,
+ portable_options.environment_config)
+
+ @staticmethod
+ def _resolve_environment(environment_type, environment_config):
environment_urn = common_urns.environments.DOCKER.urn
- if portable_options.environment_type == 'DOCKER':
+ if environment_type == 'DOCKER':
environment_urn = common_urns.environments.DOCKER.urn
- elif portable_options.environment_type == 'PROCESS':
+ elif environment_type == 'PROCESS':
environment_urn = common_urns.environments.PROCESS.urn
- elif portable_options.environment_type in ('EXTERNAL', 'LOOPBACK'):
+ elif environment_type in ('EXTERNAL', 'LOOPBACK'):
environment_urn = common_urns.environments.EXTERNAL.urn
- elif portable_options.environment_type:
- if portable_options.environment_type.startswith('beam:env:'):
- environment_urn = portable_options.environment_type
+ elif environment_type == 'EMBEDDED':
+ environment_urn = environment_type
+ elif environment_type:
+ # Allow embedded environment for testing
+ if environment_type.startswith('beam:env:') or \
+ environment_type == 'EMBEDDED':
Review comment:
Here we seem to be embedding knowledge about Java's understanding about the
special string "EMBEDDED" meaning "java embedded." There's also a python
"embedded" that is separate from this.
https://github.com/apache/beam/blob/release-2.13.0/sdks/python/apache_beam/portability/python_urns.py#L38
On the other hand, specifying "DOCKER" would give the Python docker image,
right? Seems we should be doing something more principled here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 246766)
Time Spent: 1.5h (was: 1h 20m)
> Support custom environment types when expanding external transforms
> -------------------------------------------------------------------
>
> Key: BEAM-7084
> URL: https://issues.apache.org/jira/browse/BEAM-7084
> Project: Beam
> Issue Type: Bug
> Components: beam-model, sdk-java-core
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Labels: portability
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The {{ExpansionService}} registers a default Docker JAVA SDK environment.
> Users should be able to specify other types of environment, i.e.
> Process/External/Embedded.
> We can add an {{environment_type}} field to the {{ExpansionRequest}} Proto
> message.
> CC [~robertwb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
