[
https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000297#comment-17000297
]
Alan Myrvold edited comment on BEAM-8195 at 12/19/19 6:48 PM:
--------------------------------------------------------------
I also see inconsistent retry behavior between the Python SDK and Java SDK.
For 429 errors, Python retries 4 times and Java retries 5 times.
For 503 errors, Python retries 19 times and Java retries 5 times.
Python's submit_job_description uses the default retry_on_server_errors_filter
which does not retry on 4xx errors.
Not sure where the extra python retry's are happening, but likely on the http
transport level.
To verify, ran:
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1
-N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}}
or
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1
-N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}}
Locally, then ran dataflow pipeline with
\-\-dataflow_endpoint=http://localhost:1500 (Python) or
--dataflowEndpoint=http://localhost:1500 (java)
was (Author: alanmyrvold):
I also see inconsistent retry behavior between the Python SDK and Java SDK.
For 429 errors, Python retries 4 times and Java retries 5 times.
For 503 errors, Python retries 19 times and Java retries 5 times.
Python's submit_job_description uses the default retry_on_server_errors_filter
which does not retry on 4xx errors.
Not sure where the extra python retry's are happening, but likely on the http
transport level.
To verify, ran:
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 503 Message\n" | nc -q 1
-N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}}
or
{{a=0; while true ; do a=$((a+1)); echo -e "HTTP/1.1 429 Message\n" | nc -q 1
-N -l -p 1500 > /dev/null ; echo "req $a $(date)"; done}}
Locally, then ran dataflow pipeline with
{{--dataflow_endpoint=http://localhost:1500 }}(Python) or
{{--dataflowEndpoint=http://localhost:1500 }}(java)
> Quota exceeded for create requests
> ----------------------------------
>
> Key: BEAM-8195
> URL: https://issues.apache.org/jira/browse/BEAM-8195
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core, testing
> Reporter: Ahmet Altay
> Assignee: Yifan Zou
> Priority: Critical
>
> Post commits failied with the following error:
> HttpError accessing
> <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>:
> response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type':
> 'application/json; charset=UTF-8', 'content-length': '598',
> 'transfer-encoding': 'chunked', 'cache-control': 'private',
> 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary':
> 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status':
> '429', 'x-content-type-options': 'nosniff'}>, content <{
> "error": {
> "code": 429,
> "message": "Quota exceeded for quota metric
> 'dataflow.googleapis.com/create_requests' and limit
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for
> consumer 'project_number:844138762903'.",
> "status": "RESOURCE_EXHAUSTED",
> "details": [
> {
> Could we increase the quota?
> /cc [~alanmyrvold] [~kenn]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
