[ https://issues.apache.org/jira/browse/AIRFLOW-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951272#comment-16951272 ]
ASF subversion and git services commented on AIRFLOW-4965: ---------------------------------------------------------- Commit f6cb11fb044a107397dd11262cfaf1ccaa622d41 in airflow's branch refs/heads/v1-10-stable from Kamil Breguła [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f6cb11f ] [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) * [AIRFLOW-4965] Handle quote exceptions in GCP operators > Handling throttling in GCP AI operators > --------------------------------------- > > Key: AIRFLOW-4965 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4965 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp > Affects Versions: 1.10.3 > Reporter: Kamil > Assignee: Kamil > Priority: Minor > Fix For: 1.10.6 > > > Polidea develops Apache Airflow operators for following Google Cloud AI > services: > * Cloud Translate > * Cloud Vision > * Cloud Text-To-Speech > * Cloud Speech-To-Text > * Cloud Translate Speech > * Cloud Natural Language > * Cloud Video Intelligence > Those API implement quota verification and throttle requests that exceed the > quota. Here are the relevant links describing it: > [https://cloud.google.com/translate/quotas] > [https://cloud.google.com/vision/quotas] > [https://cloud.google.com/speech-to-text/quotas] > [https://cloud.google.com/text-to-speech/quotas] > [https://cloud.google.com/natural-language/quotas] > [https://cloud.google.com/video-intelligence/quotas] > There are several types of quotas and limits: > > *Translate:* > * characters per day [403 - error “Daily Limit Exceeded”] > * characters per 100 seconds (per project or per project/user) [403 error > “User Rate Limit Exceeded”] [TEMPORARY] > > *Vision:* > * image file size > * requests per minute [TEMPORARY] > * images per feature per month > > *Text to speech:* > * total characters per request > * requests per minute [TEMPORARY] > * characters per minute [TEMPORARY] > > *Speech to text* > * limits of the content size > * limits of the phrases/characters per request for context > * requests per 60 seconds [TEMPORARY] > * processing per day > *Natural Language* > * Text Content size > * Token quota and Entity mentions (ignored?) > * requests per 100 seconds [TEMPORARY] > * requests per day > *Video Intelligence* > * video size > * requests per minute [TEMPORARY] > * backend time in seconds per minute [TEMPORARY] > > In all Cloud AI operators we are using Python Client API. Most methods are > using built-in > [Retry|https://googleapis.github.io/google-cloud-python/latest/core/retry.html#google.api_core.retry.Retry] > object and Retry mechanism. The assumption is that for functions that use > the mechanism, it is implemented correctly and by default “retriable” errors > only are retried. User can configure behaviour of the Retry object - > exponential back-off factor, delays, etc. In the current API Retry object can > be provided by the user creating the DAG and using the operator: > > The APIS that use Retry object are: > * *Cloud Vision Product Search* > * *Cloud Vision Extra* > * *Cloud Vision Detect* > * *Cloud Natural Language* > * *Cloud Speech* > * *Cloud Video Intelligence* > ** > The Retry mechanism provided by the Client API should be enough to handle > temporary bursts of requests. User can control exponential back-off rate and > will be able to adjust it to their own needs. They are also able to manually > restart failed jobs using standard Airflow mechanisms in case their > configuration is not well adjusted to their limits. > > The only case where Retry is not used in the API is Translate operator - > specifically > [translate|https://googleapis.github.io/google-cloud-python/latest/translate/client.html#google.cloud.translate_v2.client.Client.translate] > API. > > In case of Translate API, the proposal is to use Retry decorator in our own > hook and perform retries only in case of *“User Rate Limit Exceeded”* error, > all other errors (size limit and Daily Limit Exceeded) should be treated as > non-retriable. In those cases users will be able to manually restart failed > jobs. > > h1. Implementation > We analyzed two solutions: > # extension of the built-in mechanism from google-cloud-python library - > Retry > # external library - tenacity > > The use of the first solution seems natural, but it is problematic. Each > method creates a retry object by default from a configuration based on a > private file with configuration. > Reference: > > [https://github.com/googleapis/google-cloud-python/blob/b718d2d9bb32b0e7934ae90d57dc80c81ce0fb73/vision/google/cloud/vision_v1/gapic/image_annotator_client.py#L162-L168 > ] > [https://github.com/googleapis/google-cloud-python/blob/b718d2d9bb32b0e7934ae90d57dc80c81ce0fb73/vision/google/cloud/vision_v1/gapic/image_annotator_client.py#L296-L304 > ]If we would like to extend this mechanism, we would have to copy the logic > to this configuration. The google-cloud-python library does not allow us to > easily change only part of the configuration of retry object. > > The retry mechanism is not supported by all services (See: [Current > approach|https://docs.google.com/document/d/1QYwTy6r7bbLK3cmE9x1VShelHQ4B9SLHZDVjp6ja1S4/edit#heading=h.y76gaxcevnym]), > so there is a need to create a separate mechanism. The new mechanism based > on the external library will work with all services. This will provide a more > predictable developer experience. > > The tenacity library provides a code retry mechanism based on the decorator. > It use wait strategy that applies exponential backoff. All hook methods that > are covered by quota restrictions will get a new decorator. > > Sample implementation: > {{@tenacity.retry(}} > {{ wait=tenacity.wait_exponential(min=1, max=100),}} > {{ retry=retry_if_temporary_quota(),}} > {{)}} > {{def fetch():}} > {{ response = client.translate(TEXT, > target_language="PL")['translatedText']}} > {{ return response|}} > > > _retry_if_temporary_quota_ is a factory method that creates a predicate to > check if the exception concerns the quota restriction. -- This message was sent by Atlassian Jira (v8.3.4#803005)