[
https://issues.apache.org/jira/browse/BEAM-5440?focusedWorklogId=272381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-272381
]
ASF GitHub Bot logged work on BEAM-5440:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Jul/19 22:33
Start Date: 04/Jul/19 22:33
Worklog Time Spent: 10m
Work Description: sambvfx commented on pull request #8982: [BEAM-5440]
Pass docker run options to SDK harness containers
URL: https://github.com/apache/beam/pull/8982#discussion_r300506366
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
##########
@@ -1235,26 +1235,32 @@ class DockerSdkWorkerHandler(GrpcWorkerHandler):
def __init__(self, payload, state, provision_info):
super(DockerSdkWorkerHandler, self).__init__(state, provision_info)
self._container_image = payload.container_image
+ self._options = payload.options
self._container_id = None
def start_worker(self):
try:
subprocess.check_call(['docker', 'pull', self._container_image])
except Exception:
logging.info('Unable to pull image %s' % self._container_image)
- self._container_id = subprocess.check_output(
- ['docker',
- 'run',
- '-d',
- # TODO: credentials
- '--network=host',
- self._container_image,
- '--id=%s' % uuid.uuid4(),
- '--logging_endpoint=%s' % self.logging_api_service_descriptor().url,
- '--control_endpoint=%s' % self.control_address,
- '--artifact_endpoint=%s' % self.control_address,
- '--provision_endpoint=%s' % self.control_address,
- ]).strip()
+ cmd = [
+ 'docker',
+ 'run',
+ '-d',
+ # TODO: credentials
+ '--network=host',
+ ]
+ if self._options:
+ cmd.extend(self._options.split(' '))
+ cmd.extend([
+ self._container_image,
+ '--id=%s' % uuid.uuid4(),
+ '--logging_endpoint=%s' % self.logging_api_service_descriptor().url,
+ '--control_endpoint=%s' % self.control_address,
+ '--artifact_endpoint=%s' % self.control_address,
+ '--provision_endpoint=%s' % self.control_address,
+ ])
+ self._container_id = subprocess.check_output(cmd).strip()
Review comment:
The reformatting was necessary to add the optional `DockerPayload.options`
into the docker run command.
```python
if self._options:
cmd.extend(self._options.split(' '))
```
I didn't see anywhere this class was being used... I'm happy to revert if we
don't want/need this functionality here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 272381)
Time Spent: 1h (was: 50m)
> Add option to mount a directory inside SDK harness containers
> -------------------------------------------------------------
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
> Issue Type: New Feature
> Components: java-fn-execution, sdk-java-core
> Reporter: Maximilian Michels
> Priority: Major
> Labels: portability, portability-flink
> Time Spent: 1h
> Remaining Estimate: 0h
>
> While experimenting with the Python SDK locally, I found it inconvenient that
> I can't mount a host directory to the Docker containers, i.e. the input must
> already be in the container and the results of a Write remain inside the
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information,
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general
> Docker arguments field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
