[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354764 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 23:07 Start Date: 05/Dec/19 23:07 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354764) Time Spent: 3.5h (was: 3h 20m) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354763 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 23:06 Start Date: 05/Dec/19 23:06 Worklog Time Spent: 10m Work Description: robertwb commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-562358630 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354763) Time Spent: 3h 20m (was: 3h 10m) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354579=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354579 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 18:51 Start Date: 05/Dec/19 18:51 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-562263787 @mxm @robertwb tests have completed successfully. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354579) Time Spent: 3h 10m (was: 3h) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354466 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 17:03 Start Date: 05/Dec/19 17:03 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-562220292 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354466) Time Spent: 3h (was: 2h 50m) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354215 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 11:54 Start Date: 05/Dec/19 11:54 Worklog Time Spent: 10m Work Description: mxm commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-562097731 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354215) Time Spent: 2h 50m (was: 2h 40m) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=354064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-354064 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 05/Dec/19 06:22 Start Date: 05/Dec/19 06:22 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-561989079 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 354064) Time Spent: 2h 40m (was: 2.5h) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=353882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353882
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 04/Dec/19 23:18
Start Date: 04/Dec/19 23:18
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r354037327
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
@mxm updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 353882)
Time Spent: 2.5h (was: 2h 20m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=349189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349189
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 25/Nov/19 15:33
Start Date: 25/Nov/19 15:33
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10161: [BEAM-8746] Make
local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r350255347
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
Let me know when you update the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349189)
Time Spent: 2h 20m (was: 2h 10m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=349188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349188
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 25/Nov/19 15:32
Start Date: 25/Nov/19 15:32
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10161: [BEAM-8746] Make
local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r350254951
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
Yes, that looks good!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 349188)
Time Spent: 2h 10m (was: 2h)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=348220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-348220
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 22/Nov/19 17:14
Start Date: 22/Nov/19 17:14
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r349704203
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
```python
def get_bind_address(self):
"""Return the address used to open the port on the gRPC server.
"""
return self.get_connect_address()
def get_connect_address(self):
"""Return the host name at which this server will be accessible.
In particular, this is provided to the client upon connection as the
artifact staging endpoint.
"""
return 'localhost'
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
port = self._server.add_insecure_port('%s:%d' %
(self.get_bind_address(), port))
beam_job_api_pb2_grpc.add_JobServiceServicer_to_server(self,
self._server)
beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
self._artifact_service, self._server)
hostname = self.get_connect_address()
self._artifact_staging_endpoint = endpoints_pb2.ApiServiceDescriptor(
url='%s:%d' % (hostname, port))
self._server.start()
_LOGGER.info('Grpc server started at %s on port %d' % (hostname, port))
return port
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 348220)
Time Spent: 2h (was: 1h 50m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=348219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-348219
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 22/Nov/19 17:12
Start Date: 22/Nov/19 17:12
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r349703328
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
Yeah, that's probably the most future-proof and versatile solution.
What do you think about terminology for this?
1. `get_bind_address()`, `get_connect_address()`
2. `get_bind_address()`, `get_service_address()`
3. `get_bind_address()`, `get_hostname()`
Something else?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 348219)
Time Spent: 1h 50m (was: 1h 40m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=348037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-348037
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 22/Nov/19 09:56
Start Date: 22/Nov/19 09:56
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10161: [BEAM-8746] Make
local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r349514933
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
It looks ok, but one idea:
>there are two separate hostnames required, one for opening the port for the
server, and one which is delivered to the client for reconnecting to the
staging service.
Wouldn't it work to allow configuration of a `bind` address (e.g. `[::]`)
and a `connect` address (e.g. `service_name`, what is currently returned in
`get_hostname()`)? That way we would give the subclass full control over what
it wants to do. Maybe return this as a tuple to avoid multiple overrides.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 348037)
Time Spent: 1h 40m (was: 1.5h)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=347918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-347918
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 22/Nov/19 05:00
Start Date: 22/Nov/19 05:00
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r349433786
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
ok, I've been playing around with this, and the main impediment to keeping
this design simple is that there are two separate hostnames required, one for
opening the port for the server, and one which is delivered to the client for
reconnecting to the staging service.
I think it'd be nice to prevent people from having to figure out all this
stuff again, because it's pretty frustrating to get it right, so here's my best
effort at a compromise between making this configurable and making it "just
work".
```python
def get_hostname(self):
"""Return the host name at which this server will be accessible.
In particular, this is provided to the client as the
artifact staging endpoint.
"""
return 'localhost'
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
hostname = self.get_hostname()
# either open this up to the world, or lock it down to localhost
if os.environ.get('DOCKER_MAC_CONTAINER') == '1' or hostname !=
'localhost':
service_address = '[::]'
else:
service_address = 'localhost'
port = self._server.add_insecure_port('%s:%d' % (service_address, port))
beam_job_api_pb2_grpc.add_JobServiceServicer_to_server(self,
self._server)
beam_artifact_api_pb2_grpc.add_ArtifactStagingServiceServicer_to_server(
self._artifact_service, self._server)
self._artifact_staging_endpoint = endpoints_pb2.ApiServiceDescriptor(
url='%s:%d' % (hostname, port))
self._server.start()
_LOGGER.info('Grpc server started at %s on port %d' % (hostname, port))
return port
```
What do you think? The other option is that I just copy all of
`start_grpc_server` into my sub-class. It's not the end of the world if that's
the decision we come to.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 347918)
Time Spent: 1.5h (was: 1h 20m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=347126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-347126
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 21/Nov/19 02:10
Start Date: 21/Nov/19 02:10
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r348866533
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
I think it'd make sense for this to be parameterized, likely with localhost
as a default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 347126)
Time Spent: 1h 20m (was: 1h 10m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=347032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-347032
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 20/Nov/19 22:14
Start Date: 20/Nov/19 22:14
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r348773235
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
> Could this not be handled in the subclass?
Yeah, let me look into the best design for this. It'd be nice if we took
this opportunity to make more than the hostname configurable (i.e. provide a
way to further configure the server before starting). I'll propose an
alternative later today.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 347032)
Time Spent: 1h 10m (was: 1h)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=347003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-347003
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 20/Nov/19 21:01
Start Date: 20/Nov/19 21:01
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10161: [BEAM-8746] Make
local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r348742690
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
Could this not be handled in the subclass? I think the notion of the
`LocalJobServer` is not to listen on all interfaces. We could make the bind
address configurable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 347003)
Time Spent: 1h (was: 50m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=346835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346835
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 20/Nov/19 16:11
Start Date: 20/Nov/19 16:11
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10161: [BEAM-8746]
Make local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r348575097
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
A slightly new use case. Instead of throwing away the job server for each
invocation, we'd like to keep it around for multiple jobs, like we do with the
Flink job server, then we can use our job-server-ui to view and manage jobs
there.
Additionally, we are sub-classing this class (`LocalJobServicer`) to make a
new kind of job server that is able to proxy other job servers - a middle-man
for reviewing a job before sending it on to Flink, Direct, Spark, etc.
As a result we want to stand this service up along with our other
long-running Beam services.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 346835)
Time Spent: 50m (was: 40m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=346834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346834 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 20/Nov/19 16:06 Start Date: 20/Nov/19 16:06 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-556075520 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 346834) Time Spent: 40m (was: 0.5h) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[
https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=346652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346652
]
ASF GitHub Bot logged work on BEAM-8746:
Author: ASF GitHub Bot
Created on: 20/Nov/19 12:04
Start Date: 20/Nov/19 12:04
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10161: [BEAM-8746] Make
local job service accessible from external machines
URL: https://github.com/apache/beam/pull/10161#discussion_r348443690
##
File path: sdks/python/apache_beam/runners/portability/local_job_service.py
##
@@ -95,7 +95,7 @@ def create_beam_job(self, preparation_id, job_name,
pipeline, options):
def start_grpc_server(self, port=0):
self._server = grpc.server(UnboundedThreadPoolExecutor())
-port = self._server.add_insecure_port('localhost:%d' % port)
+port = self._server.add_insecure_port('[::]:%d' % port)
Review comment:
Is this for a new use case? Originally, the embedded job server is intended
for testing pipeline execution on the same machine or container. `localhost`
should work fine for this case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 346652)
Time Spent: 0.5h (was: 20m)
> Allow the local job service to work from inside docker
> --
>
> Key: BEAM-8746
> URL: https://issues.apache.org/jira/browse/BEAM-8746
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently the connection is refused. It's a simple fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=346349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346349 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 19/Nov/19 23:26 Start Date: 19/Nov/19 23:26 Worklog Time Spent: 10m Work Description: chadrik commented on pull request #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161 Simple fix that makes it possible to access a local job service running in docker. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8746) Allow the local job service to work from inside docker
[ https://issues.apache.org/jira/browse/BEAM-8746?focusedWorklogId=346350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-346350 ] ASF GitHub Bot logged work on BEAM-8746: Author: ASF GitHub Bot Created on: 19/Nov/19 23:26 Start Date: 19/Nov/19 23:26 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10161: [BEAM-8746] Make local job service accessible from external machines URL: https://github.com/apache/beam/pull/10161#issuecomment-555763150 R: @mxm R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 346350) Time Spent: 20m (was: 10m) > Allow the local job service to work from inside docker > -- > > Key: BEAM-8746 > URL: https://issues.apache.org/jira/browse/BEAM-8746 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently the connection is refused. It's a simple fix. -- This message was sent by Atlassian Jira (v8.3.4#803005)
