[
https://issues.apache.org/jira/browse/BEAM-11935?focusedWorklogId=570559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-570559
]
ASF GitHub Bot logged work on BEAM-11935:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Mar/21 15:39
Start Date: 23/Mar/21 15:39
Worklog Time Spent: 10m
Work Description: chamikaramj commented on a change in pull request
#14189:
URL: https://github.com/apache/beam/pull/14189#discussion_r599687746
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -740,6 +719,25 @@ def _apply_sdk_environment_overrides(
new_payload.container_image = new_container_image
environment.payload = new_payload.SerializeToString()
+ # De-dup environments by Docker container image since currently Dataflow
Review comment:
To clarify, the restriction is already there for Dataflow. We currently
start an SDK Harness per container image and de-dup here:
https://github.com/apache/beam/blob/83bd5485047373ae0e380c54063e3769874a8b09/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L304
This just moves the de-duping logic from container images started to
Dataflow to environments in the proto since I'm trying to update Dataflow to
map work items to environments based on the environment ID (not container
image).
I can try to reduce de-duping to multi-language Java environments in
muti-language pipelines since multiple Python environments do not seem to be
running into issues currently. Multiple Java environments in multi-language
pipelines run into dependency conflicts. Does that help ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 570559)
Time Spent: 3h (was: 2h 50m)
> Update Dataflow requests to include the environment ID
> ------------------------------------------------------
>
> Key: BEAM-11935
> URL: https://issues.apache.org/jira/browse/BEAM-11935
> Project: Beam
> Issue Type: Bug
> Components: cross-language, runner-dataflow
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P2
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
