[jira] [Commented] (AIRFLOW-5544) GCP datastore operation pull should have some retry to avoid false alarm

[Kamil Bregula (Jira)]

    [ 
https://issues.apache.org/jira/browse/AIRFLOW-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982996#comment-16982996
 ] 

Kamil Bregula commented on AIRFLOW-5544:
----------------------------------------

There is a plan to make these operators available for Cloud Composer in the 
near future. 
[https://lists.apache.org/thread.html/2c9559184045e772acd21cbdd7435f6bf89c76eb9311311d58d16e5f@%3Cdev.airflow.apache.org%3E]

You will be able to install the latest operators supported by my team.

> GCP datastore operation pull should have some retry to avoid false alarm
> ------------------------------------------------------------------------
>
>                 Key: AIRFLOW-5544
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5544
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: gcp
>    Affects Versions: 1.10.5
>            Reporter: Yang Hu
>            Priority: Minor
>
> Hi Airflow,
> We noticed that sometime, google status url will return us 500 and airflow 
> will mark the whole task failed, but is actually runs fine.
>  
> I think we need some retry before failing the task, at least at this 500 case
> Sample request:
> > [2019-09-18 21:45:19,543] \{base_task_runner.py:98} INFO - Subtask: 
> googleapiclient.errors.HttpError: &lt;HttpError 500 when requesting <a 
> href="https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json";
>  
> target="_blank">https://datastore.googleapis.com/v1/projects/MASKED/operations/AS_MASKED_RI?alt=json</a>
>  returned &quot;Unknown Error.&quot;&gt;
> Ref code:
> [https://github.com/apache/airflow/blob/7e4330cce0b4f333b1658cdc315b06505cf9dd76/airflow/gcp/hooks/datastore.py#L269]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)