[
https://issues.apache.org/jira/browse/BEAM-8507?focusedWorklogId=335611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-335611
]
ASF GitHub Bot logged work on BEAM-8507:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Oct/19 16:44
Start Date: 29/Oct/19 16:44
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #9918: [BEAM-8507] Fix
'flink_master' parameter for the upcoming release
URL: https://github.com/apache/beam/pull/9918#discussion_r340194245
##########
File path: sdks/python/apache_beam/runners/portability/flink_runner.py
##########
@@ -32,26 +33,39 @@
class FlinkRunner(portable_runner.PortableRunner):
def default_job_server(self, options):
- flink_master = options.view_as(FlinkRunnerOptions).flink_master
- if flink_master == '[local]' or sys.version_info < (3, 6):
+ flink_master = self.add_http_scheme(
+ options.view_as(FlinkRunnerOptions).flink_master)
+ options.view_as(FlinkRunnerOptions).flink_master = flink_master
+ if flink_master in ['[local]', '[auto]'] or sys.version_info < (3, 6):
portable_options = options.view_as(pipeline_options.PortableOptions)
if flink_master == '[local]' and not portable_options.environment_type:
- portable_options.environment_type == 'LOOPBACK'
+ portable_options.environment_type = 'LOOPBACK'
return job_server.StopOnExitJobServer(FlinkJarJobServer(options))
else:
return flink_uber_jar_job_server.FlinkUberJarJobServer(flink_master)
+ @staticmethod
+ def add_http_scheme(flink_master):
+ flink_master = flink_master.strip()
+ # Add http protocol scheme if not provided
+ if not flink_master in ['[local]', '[auto]'] and \
+ not flink_master.startswith('http://'):
+ logging.info('Adding HTTP protocol scheme to flink_master parameter: '
+ 'http://%s', flink_master)
+ flink_master = 'http://' + flink_master
+ return flink_master
+
class FlinkRunnerOptions(pipeline_options.PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
parser.add_argument('--flink_master',
- default='[auto]',
- help='Flink master address (host:port) to submit the'
- ' job against. Use "[local]" to start a local'
- ' cluster for the execution. Use "[auto]" if you'
- ' plan to either execute locally or submit
through'
- ' Flink\'s CLI.')
+ default='[local]',
Review comment:
+1 I think these changes make sense. Although I would definitely like to see
some tests (which I guess is
[BEAM-8512](https://issues.apache.org/jira/browse/BEAM-8512))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 335611)
Time Spent: 2h (was: 1h 50m)
> Unspecified flink_master fails on Python 3.6+
> ---------------------------------------------
>
> Key: BEAM-8507
> URL: https://issues.apache.org/jira/browse/BEAM-8507
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Kyle Weaver
> Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> [auto] was made the default flink_master in
> [https://github.com/apache/beam/pull/9844]. Since FlinkUberJarJobServer does
> not support [auto], this is a problem for Python 3.6 and 3.7. (It is fine for
> 3.5 because 3.5 does not use FlinkUberJarJobServer.)
>
> requests.exceptions.InvalidURL: Failed to parse: [auto]/v1/config
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
