[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313260 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 19:55 Start Date: 16/Sep/19 19:55 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531932686 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313260) Time Spent: 2h 50m (was: 2h 40m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313258 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 19:44 Start Date: 16/Sep/19 19:44 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531928609 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313258) Time Spent: 2h 40m (was: 2.5h) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 40m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313243 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 18:39 Start Date: 16/Sep/19 18:39 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531904051 Sorry for the lack of Python-fu. Fixed up and now passing locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313243) Time Spent: 2.5h (was: 2h 20m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313231 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 18:15 Start Date: 16/Sep/19 18:15 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#discussion_r324815390 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -383,7 +383,7 @@ def run_pipeline(self, pipeline, options): use_fnapi = apiclient._use_fnapi(options) from apache_beam.portability.api import beam_runner_api_pb2 default_container_image = ( Review comment: That makes sense. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313231) Time Spent: 2h 20m (was: 2h 10m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 20m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313210 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:46 Start Date: 16/Sep/19 17:46 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531883420 LGTM as long as tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313210) Time Spent: 2h 10m (was: 2h) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313202 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:38 Start Date: 16/Sep/19 17:38 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531880508 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313202) Time Spent: 2h (was: 1h 50m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313185 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:12 Start Date: 16/Sep/19 17:12 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#discussion_r324788086 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -908,15 +904,18 @@ def _get_container_image_tag(): return base_version -def get_default_container_image_for_current_sdk(use_fnapi): +def get_container_image_from_options(pipeline_options): """For internal use only; no backwards-compatibility guarantees. Args: - use_fnapi (bool): True, if pipeline is using FnAPI, False otherwise. + pipeline_options (PipelineOptions): A container for pipeline options. Returns: - str: Google Cloud Dataflow container image for remote execution. + str: Container image for remote execution. """ + if pipeline_options.As(WorkerOptions).worker_harness_container_image: Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313185) Time Spent: 1h 50m (was: 1h 40m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1h 50m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313184 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:10 Start Date: 16/Sep/19 17:10 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#discussion_r324787316 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -383,7 +383,7 @@ def run_pipeline(self, pipeline, options): use_fnapi = apiclient._use_fnapi(options) from apache_beam.portability.api import beam_runner_api_pb2 default_container_image = ( Review comment: I think it still is considered the "default" for the context of this pipeline. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313184) Time Spent: 1h 40m (was: 1.5h) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313183 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:09 Start Date: 16/Sep/19 17:09 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#discussion_r324787117 ## File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py ## @@ -908,15 +904,18 @@ def _get_container_image_tag(): return base_version -def get_default_container_image_for_current_sdk(use_fnapi): +def get_container_image_from_options(pipeline_options): """For internal use only; no backwards-compatibility guarantees. Args: - use_fnapi (bool): True, if pipeline is using FnAPI, False otherwise. + pipeline_options (PipelineOptions): A container for pipeline options. Returns: - str: Google Cloud Dataflow container image for remote execution. + str: Container image for remote execution. """ + if pipeline_options.As(WorkerOptions).worker_harness_container_image: Review comment: I think correct syntax is `pipeline_options.view_as(WorkerOptions)`. Does current syntax work? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313183) Time Spent: 1.5h (was: 1h 20m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313182
]
ASF GitHub Bot logged work on BEAM-8240:
Author: ASF GitHub Bot
Created on: 16/Sep/19 17:09
Start Date: 16/Sep/19 17:09
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9583: [BEAM-8240]
Ensure that the pipeline proto contains the worker_harness_container_image
override
URL: https://github.com/apache/beam/pull/9583#discussion_r324786755
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -176,6 +176,19 @@ def test_create_runner(self):
isinstance(create_runner('TestDataflowRunner'),
TestDataflowRunner))
+ def test_environment_override_translation(self):
+self.default_properties.append("--experiments=beam_fn_api")
+self.default_properties.append("--worker_harness_container_image=FOO")
+remote_runner = DataflowRunner()
+p = Pipeline(remote_runner,
+ options=PipelineOptions(self.default_properties))
+(p | ptransform.Create([1, 2, 3]) # pylint:
disable=expression-not-assigned
+ | 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
+ | ptransform.GroupByKey())
+p.run()
+pipeline_proto = remote_runner.pipeline_proto
+self.assertEqual(pipeline_proto.components.envirnoments.values(), ["FOO"])
Review comment:
It was a hastily made test, was relying on jenkins to test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313182)
Time Spent: 1h 20m (was: 1h 10m)
> SDK Harness
> ---
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container
> within environment field when building pipeline proto.
>
> Passing in the experiment *worker_harness_container_image=YYY* doesn't
> override the pipeline proto environment field and it is still being populated
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313177 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:05 Start Date: 16/Sep/19 17:05 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#discussion_r324785305 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -383,7 +383,7 @@ def run_pipeline(self, pipeline, options): use_fnapi = apiclient._use_fnapi(options) from apache_beam.portability.api import beam_runner_api_pb2 default_container_image = ( Review comment: Maybe do not call this variable "default"_container_image any more, since it not really just default. Alternatively, we can remove the variable and change L390 to read `container_image=apiclient.get_container_image_from_options(options)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313177) Time Spent: 1h 10m (was: 1h) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313173=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313173
]
ASF GitHub Bot logged work on BEAM-8240:
Author: ASF GitHub Bot
Created on: 16/Sep/19 17:03
Start Date: 16/Sep/19 17:03
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #9583:
[BEAM-8240] Ensure that the pipeline proto contains the
worker_harness_container_image override
URL: https://github.com/apache/beam/pull/9583#discussion_r324784428
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -176,6 +176,19 @@ def test_create_runner(self):
isinstance(create_runner('TestDataflowRunner'),
TestDataflowRunner))
+ def test_environment_override_translation(self):
+self.default_properties.append("--experiments=beam_fn_api")
+self.default_properties.append("--worker_harness_container_image=FOO")
+remote_runner = DataflowRunner()
+p = Pipeline(remote_runner,
+ options=PipelineOptions(self.default_properties))
+(p | ptransform.Create([1, 2, 3]) # pylint:
disable=expression-not-assigned
+ | 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
+ | ptransform.GroupByKey())
+p.run()
+pipeline_proto = remote_runner.pipeline_proto
+self.assertEqual(pipeline_proto.components.envirnoments.values(), ["FOO"])
Review comment:
Consider adding a comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313173)
Time Spent: 1h (was: 50m)
> SDK Harness
> ---
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Time Spent: 1h
> Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container
> within environment field when building pipeline proto.
>
> Passing in the experiment *worker_harness_container_image=YYY* doesn't
> override the pipeline proto environment field and it is still being populated
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313168
]
ASF GitHub Bot logged work on BEAM-8240:
Author: ASF GitHub Bot
Created on: 16/Sep/19 17:02
Start Date: 16/Sep/19 17:02
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #9583:
[BEAM-8240] Ensure that the pipeline proto contains the
worker_harness_container_image override
URL: https://github.com/apache/beam/pull/9583#discussion_r324783913
##
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
##
@@ -176,6 +176,19 @@ def test_create_runner(self):
isinstance(create_runner('TestDataflowRunner'),
TestDataflowRunner))
+ def test_environment_override_translation(self):
+self.default_properties.append("--experiments=beam_fn_api")
+self.default_properties.append("--worker_harness_container_image=FOO")
+remote_runner = DataflowRunner()
+p = Pipeline(remote_runner,
+ options=PipelineOptions(self.default_properties))
+(p | ptransform.Create([1, 2, 3]) # pylint:
disable=expression-not-assigned
+ | 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
+ | ptransform.GroupByKey())
+p.run()
+pipeline_proto = remote_runner.pipeline_proto
+self.assertEqual(pipeline_proto.components.envirnoments.values(), ["FOO"])
Review comment:
Just to clarify, does "["FOO"]" somehow end up being equal to a
DockerPayload with "FOO" set to container_image ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 313168)
Time Spent: 50m (was: 40m)
> SDK Harness
> ---
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container
> within environment field when building pipeline proto.
>
> Passing in the experiment *worker_harness_container_image=YYY* doesn't
> override the pipeline proto environment field and it is still being populated
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313166 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 17:01 Start Date: 16/Sep/19 17:01 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531866276 Note that this fixes the dataflow runner (and not portable runner) to ensure that the JSON worker pool and environment proto match. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313166) Time Spent: 40m (was: 0.5h) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313161 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 16:57 Start Date: 16/Sep/19 16:57 Worklog Time Spent: 10m Work Description: robertwb commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531864700 Is this the flag users should be setting (rather than, say, environment_config)? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313161) Time Spent: 0.5h (was: 20m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313151 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 16:51 Start Date: 16/Sep/19 16:51 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8240) SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=313152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313152 ] ASF GitHub Bot logged work on BEAM-8240: Author: ASF GitHub Bot Created on: 16/Sep/19 16:51 Start Date: 16/Sep/19 16:51 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9583: [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override URL: https://github.com/apache/beam/pull/9583#issuecomment-531862416 R: @tvalentyn @chamikaramj CC: @ihji @ananvay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 313152) Time Spent: 20m (was: 10m) > SDK Harness > --- > > Key: BEAM-8240 > URL: https://issues.apache.org/jira/browse/BEAM-8240 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Luke Cwik >Assignee: Luke Cwik >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > SDK harness incorrectly identifies itself when using custom SDK container > within environment field when building pipeline proto. > > Passing in the experiment *worker_harness_container_image=YYY* doesn't > override the pipeline proto environment field and it is still being populated > with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802* > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
