arjunanan6 commented on issue #29318:
URL: https://github.com/apache/airflow/issues/29318#issuecomment-1475183918
@ScriptPup Sorry it took ages on this one, but life happened :)
I spent a little time looking at this today and I wanted to get a little
more context on the problem, because to me, it appears that the information is
already available in the error output.
This is from my attempt to install a non-existent requirement:
```console
[2023-03-19, 09:30:01 UTC] {task_command.py:406} INFO - Running
<TaskInstance: VENVTASK.virtualenv_classic scheduled__2023-03-19T09:20:00+00:00
[running]> on host 7007f5fd4d69
[2023-03-19, 09:30:01 UTC] {taskinstance.py:1557} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='VENVTASK'
AIRFLOW_CTX_TASK_ID='virtualenv_classic'
AIRFLOW_CTX_EXECUTION_DATE='2023-03-19T09:20:00+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='scheduled__2023-03-19T09:20:00+00:00'
[2023-03-19, 09:30:01 UTC] {process_utils.py:179} INFO - Executing cmd:
/usr/local/bin/python -m virtualenv /tmp/venv89mk8x6s --system-site-packages
[2023-03-19, 09:30:01 UTC] {process_utils.py:183} INFO - Output:
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - created virtual
environment CPython3.8.16.final.0-64 in 345ms
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - creator
CPython3Posix(dest=/tmp/venv89mk8x6s, clear=False, no_vcs_ignore=False,
global=True)
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - seeder
FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle,
via=copy, app_data_dir=/root/.local/share/virtualenv)
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - added seed
packages: pip==23.0, setuptools==67.1.0, wheel==0.38.4
[2023-03-19, 09:30:01 UTC] {process_utils.py:187} INFO - activators
BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
[2023-03-19, 09:30:01 UTC] {process_utils.py:179} INFO - Executing cmd:
/tmp/venv89mk8x6s/bin/pip install -r /tmp/venv89mk8x6s/requirements.txt
[2023-03-19, 09:30:01 UTC] {process_utils.py:183} INFO - Output:
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: Could not
find a version that satisfies the requirement colowsssrama==0.4.0 (from
versions: none)
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: No matching
distribution found for colowsssrama==0.4.0
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO -
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO - [notice] A new
release of pip is available: 23.0 -> 23.0.1
[2023-03-19, 09:30:03 UTC] {process_utils.py:187} INFO - [notice] To update,
run: python -m pip install --upgrade pip
[2023-03-19, 09:30:03 UTC] {taskinstance.py:1837} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/operators/python.py", line 356, in execute
return super().execute(context=serializable_context)
File "/opt/airflow/airflow/operators/python.py", line 176, in execute
return_value = self.execute_callable()
File "/opt/airflow/airflow/operators/python.py", line 540, in
execute_callable
prepare_virtualenv(
File "/opt/airflow/airflow/utils/python_virtualenv.py", line 99, in
prepare_virtualenv
execute_in_subprocess(pip_cmd)
File "/opt/airflow/airflow/utils/process_utils.py", line 168, in
execute_in_subprocess
execute_in_subprocess_with_kwargs(cmd, cwd=cwd)
File "/opt/airflow/airflow/utils/process_utils.py", line 191, in
execute_in_subprocess_with_kwargs
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venv89mk8x6s/bin/pip',
'install', '-r', '/tmp/venv89mk8x6s/requirements.txt']' returned non-zero exit
status 1.
```
As you see here, before the output from CalledProcessError, there is
sufficient INFO level logging available for the things that did go wrong:
```console
[2023-03-19, 09:30:02 UTC] {process_utils.py:187} INFO - ERROR: No matching
distribution found for colowsssrama==0.4.0
```
From what I am looking at, your requirement is already implemented in
process_utils. If you have multiple requirements, then it is a bit tricky
because it fails at the very first one which could not be installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
