JavierLopezT opened a new issue #17546:
URL: https://github.com/apache/airflow/issues/17546
I am getting this error in a lot of DAGs that seem very random. The full
trace of the error is:
```
Traceback (most recent call last):
[2021-08-11 09:29:19,497] INFO - File
"/home/***/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 18, in
<module>
[2021-08-11 09:29:19,498] INFO - from . import _mysql
[2021-08-11 09:29:19,499] INFO - ImportError:
/usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS
block
[2021-08-11 09:29:19,500] INFO -
[2021-08-11 09:29:19,501] INFO - During handling of the above exception,
another exception occurred:
[2021-08-11 09:29:19,502] INFO -
[2021-08-11 09:29:19,503] INFO - Traceback (most recent call last):
[2021-08-11 09:29:19,504] INFO - File
"/opt/***/dags/import_apple_app_store/py_s3_apple.py", line 7, in <module>
[2021-08-11 09:29:19,505] INFO - from ***.models import Variable
[2021-08-11 09:29:19,506] INFO - File
"/home/***/.local/lib/python3.8/site-packages/***/__init__.py", line 46, in
<module>
[2021-08-11 09:29:19,506] INFO - settings.initialize()
[2021-08-11 09:29:19,507] INFO - File
"/home/***/.local/lib/python3.8/site-packages/***/settings.py", line 445, in
initialize
[2021-08-11 09:29:19,508] INFO - configure_adapters()
[2021-08-11 09:29:19,508] INFO - File
"/home/***/.local/lib/python3.8/site-packages/***/settings.py", line 325, in
configure_adapters
[2021-08-11 09:29:19,509] INFO - import MySQLdb.converters
[2021-08-11 09:29:19,510] INFO - File
"/home/***/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 24, in
<module>
[2021-08-11 09:29:19,510] INFO - version_info, _mysql.version_info,
_mysql.__file__
[2021-08-11 09:29:19,511] INFO - NameError: name '_mysql' is not defined
[2021-08-11 09:29:19,647] {python_file_operator.py:118} INFO - Command
exited with return code 1
```
Characteristics:
- Official Docker image apache/airflow:2.1.2.
- I add to the official image the following:
` RUN sudo apt-get update && sudo apt-get install -y build-essential
gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1
libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation
libappindicator1 libnss3 lsb-release xdg-utils wget libappindicator3-1 libgbm1`
- Also I have a requirements file. The only mysql-related library is the
provider one.
- Prod env in an Ubuntu 18.04 AWS EC2 of 128GB RAM
- Local env MacOS 10GB allocated to Docker. I have this parameter in the
docker-compose `AIRFLOW__OPERATORS__DEFAULT_RAM: 2048`
- All errors occurred in python scripts executed by BashOperators
- All errors happen during the imports, not even a single line of code after
the imports is executed
- Almost any DAG with the error uses any MySql library
First, in prod, with Python 3.6, I had the error on several DAGs (let's say
DAGs A and B worked, and C and D did not). In both A and B, there was a common
point, importing Variable before SnowflakeHook. So the code was:
```
from snowflake_hook import SnowflakeHook (this file does NOT make use of
Variable anywhere in the code)
from airflow.models import Variable
```
And changing the order solved the problem:
```
from airflow.models import Variable
from snowflake_hook import SnowflakeHook
```
Now, in local, I was testing updating the python version using the official
images. Starting from Python 3.9, suddenly, the dag A, which was already fixed
in prod, stopped working.
Then I change to Python 3.8 and the dag A starts working again. However, dag
C, which never had a problem, start failing with the same error. In this case,
there were no import of SnowflakeHook, just common python libraries imports.
Anyway, I change the code from:
```
import logging
import pandas as pd
import requests
from airflow.models import Variable
```
to
```
from airflow.models import Variable
import logging
import pandas as pd
import requests
```
And it works.
Furthermore, dag D, which also didn't have a problem, fails also. I try to
change the orders of the imports, and in this case, it does not work. At the
time of failing, there were only 5 dags active and just 1 of them (light one)
running, so it does not seem to be a RAM problem.
So I am very lost about how to deal with this error. The docker-compose is
very similar to the one in the official docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
