iostreamdoth edited a comment on issue #16243:
URL: https://github.com/apache/airflow/issues/16243#issuecomment-881909400
I have encountered similar error in airflow 1.10.15 (on google cloud
composer) with DataprocInstantiateWorkflowTemplateOperator. I think it has
nothing to do with airflow as is but may be with google api's or some sort of
library version mismatch. I have 17 tasks running in parallel for the above
operator. and 15/17 fail with `Task exited with return code Negsignal.SIGSEGV `
I m not able to get in more information but Here is what I did to get fewer
errors.
I override execute function of the above operator.
```
class
InstantiateDataProcWorkflowTemplate(dataproc.DataprocInstantiateWorkflowTemplateOperator):
def execute(self, context):
def callback(operation_future):
# Handle result.
result = operation_future.result()
hook = DataprocHook(gcp_conn_id=self.gcp_conn_id,
impersonation_chain=self.impersonation_chain)
self.log.info('Instantiating template %s', self.template_id)
operation = hook.instantiate_workflow_template(
project_id=self.project_id,
location=self.region,
template_name=self.template_id,
version=self.version,
request_id=self.request_id,
parameters=self.parameters,
retry=self.retry,
timeout=self.timeout,
metadata=self.metadata,
)
self.log.info('Template Started.')
while not operation.done():
self.log.info('Workflow template still running.')
time.sleep(600)
self.log.info('Template Finished.')
```
Doing the above did result in fewer task failing around 3/17 failed but the
failure exists for no substantial reason.
If I remove the while block `while not operation.done():` or
operation.result() from the original
[here](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/dataproc.html#DataprocInstantiateWorkflowTemplateOperator)
. The task runs successfully but no wait for completion. This lead me to
believe that there is some bug with asyn operation call on google.api_core. are
majority of people facing this problem using google provider operators?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
