[
https://issues.apache.org/jira/browse/BEAM-6540?focusedWorklogId=192987&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192987
]
ASF GitHub Bot logged work on BEAM-6540:
----------------------------------------
Author: ASF GitHub Bot
Created on: 31/Jan/19 18:58
Start Date: 31/Jan/19 18:58
Worklog Time Spent: 10m
Work Description: dustin12 commented on pull request #7664: [BEAM-6540]
Windmill quota piping
URL: https://github.com/apache/beam/pull/7664#discussion_r252796552
##########
File path:
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
##########
@@ -318,6 +318,18 @@ public Job translate(List<DataflowPackage> packages) {
WorkerPool workerPool = new WorkerPool();
+ // If streaming engine is enabled set the proper experiments so that it
is enabled on the
+ // back end as well.
Review comment:
Currently the dependency is one way the flag enables the experiments but the
experiments don't enable the flag. streamingEngineEnabled checks all three but
using it here would introduce a circular dependency.
> Adding dependency on module
'beam-runners-google-cloud-dataflow-java-fn-api-worker_main' will introduce
circular dependency between modules
'beam-runners-google-cloud-dataflow-java-fn-api-worker_main' and
'beam-runners-google-cloud-dataflow-java_main'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 192987)
Time Spent: 3h 20m (was: 3h 10m)
> Autoscaling should be aware of Streaming RPC Quota
> ---------------------------------------------------
>
> Key: BEAM-6540
> URL: https://issues.apache.org/jira/browse/BEAM-6540
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.11.0
> Reporter: Dustin Rhodes
> Assignee: Tyler Akidau
> Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Streaming Windmill Service introduces quota for the shared windmill workers.
> Autoscaling needs to be aware of throttling due to this quota in order to not
> upscale. This PR adds in that reporting.
>
> It also introduces the flag --EnableStreamingEngine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
