scaoupgrade commented on issue #8212:
URL: https://github.com/apache/airflow/issues/8212#issuecomment-788098857
> The issue seems to be infinite recursion due to an interaction between
gevent's monkey patching and the botocore library used by S3TaskHandler:
>
> ```
> Traceback (most recent call last):
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/airflow/utils/log/s3_task_handler.py",
line 131, in s3_log_exists
> return self.hook.get_key(remote_log_location) is not None
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/airflow/hooks/S3_hook.py",
line 224, in get_key
> obj = self.get_resource_type('s3').Object(bucket_name, key)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/airflow/contrib/hooks/aws_hook.py",
line 186, in get_resource_type
> config=config, verify=self.verify)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/boto3/session.py",
line 389, in resource
> aws_session_token=aws_session_token, config=config)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/boto3/session.py",
line 263, in client
> aws_session_token=aws_session_token, config=config)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/session.py",
line 823, in create_client
> credentials = self.get_credentials()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/session.py",
line 428, in get_credentials
> 'credential_provider').load_credentials()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/session.py",
line 919, in get_component
> self._components[name] = factory()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/session.py",
line 149, in _create_credential_resolver
> self, region_name=self._last_client_region_used
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/credentials.py",
line 70, in create_credential_resolver
> container_provider = ContainerProvider()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/credentials.py",
line 1803, in __init__
> fetcher = ContainerMetadataFetcher()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/utils.py",
line 1578, in __init__
> timeout=self.TIMEOUT_SECONDS
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/httpsession.py",
line 180, in __init__
> self._manager = PoolManager(**self._get_pool_manager_kwargs())
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/httpsession.py",
line 188, in _get_pool_manager_kwargs
> 'ssl_context': self._get_ssl_context(),
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/httpsession.py",
line 197, in _get_ssl_context
> return create_urllib3_context()
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/site-packages/botocore/httpsession.py",
line 72, in create_urllib3_context
> context.options |= options
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/ssl.py", line
507, in options
> super(SSLContext, SSLContext).options.__set__(self, value)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/ssl.py", line
507, in options
> super(SSLContext, SSLContext).options.__set__(self, value)
> File
"/usr/local/Caskroom/miniconda/base/envs/airflow/lib/python3.7/ssl.py", line
507, in options
> super(SSLContext, SSLContext).options.__set__(self, value)
> [Previous line repeated 409 more times]
> RecursionError: maximum recursion depth exceeded
> ```
>
> I still need to dive deeper into this
this is the root cause. Related to the monkey patching and import order of
request module:
https://github.com/gevent/gevent/issues/1016#issuecomment-328529454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
