taranlu-houzz opened a new issue, #49146:
URL: https://github.com/apache/airflow/issues/49146

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.8.1
   
   ### What happened?
   
   I have the following setup/env:
   - Non-trivial dag using taskflow
   - Tasks are mainly wrappers that call into an api running on Kubernetes
   - Running airflow locally for testing using `airflow standalone`
   - OS: macOS 14.7.5 (23H527)
   
   Problem:
   - When I run my dag, it will often succeed at first and then for a few 
successive runs before getting stuck on a seemly random task (various tasks, 
sometimes util tasks that are as simple as iterating a list and returning a 
tuple).
   - There are also other times where a task gets stuck on the first run 
(sometimes even the first task).
   - I have added a debug log as the first line of each task, and I can see 
that when this happens, the log does not show up.
   - This is an example of what the log output looks like when a task gets 
stuck:
   ```
   hzzMBP-JXP71DFYQ7
   *** Found local files:
   ***   * 
/Users/taran/airflow/logs/dag_id=export_convert_and_upload_updated_products/run_id=scheduled__2025-04-11T22:03:00+00:00/task_id=get_iterable_chunks/attempt=1.log
   *** Could not read served logs: [Errno 8] nodename nor servname provided, or 
not known
   [2025-04-11, 22:04:17 UTC] {taskinstance.py:1956} INFO - Dependencies all 
met for dep_context=non-requeueable deps ti=<TaskInstance: 
export_convert_and_upload_updated_products.get_iterable_chunks 
scheduled__2025-04-11T22:03:00+00:00 [queued]>
   [2025-04-11, 22:04:17 UTC] {taskinstance.py:1956} INFO - Dependencies all 
met for dep_context=requeueable deps ti=<TaskInstance: 
export_convert_and_upload_updated_products.get_iterable_chunks 
scheduled__2025-04-11T22:03:00+00:00 [queued]>
   [2025-04-11, 22:04:17 UTC] {taskinstance.py:2170} INFO - Starting attempt 1 
of 3
   [2025-04-11, 22:04:17 UTC] {taskinstance.py:2191} INFO - Executing 
<Task(_PythonDecoratedOperator): get_iterable_chunks> on 2025-04-11 
22:03:00+00:00
   [2025-04-11, 22:04:17 UTC] {standard_task_runner.py:60} INFO - Started 
process 73907 to run task
   [2025-04-11, 22:04:17 UTC] {standard_task_runner.py:87} INFO - Running: 
['airflow', 'tasks', 'run', 'export_convert_and_upload_updated_products', 
'get_iterable_chunks', 'scheduled__2025-04-11T22:03:00+00:00', '--job-id', 
'132', '--raw', '--subdir', 
'/Users/taran/tl/src/git/github.com/houzz/dw-airflow/airflow/dags/3d/pipeline3d_dags.py',
 '--cfg-path', '/var/folders/_s/t531j8b96sg3gzbpsc40s_5c0000gn/T/tmpw6190jpy']
   [2025-04-11, 22:04:17 UTC] {standard_task_runner.py:88} INFO - Job 132: 
Subtask get_iterable_chunks
   ```
   - The "Could not read served logs" error (?) seems like it might be related, 
because when that task happens to complete successfully, it does not occur.
   
   
   ### What you think should happen instead?
   
   I would expect the `execution_timeout` and `retries` params to override any 
other issues and always work. I would also not expect there to be the error 
related to logs, since everything is being run locally.
   
   ### How to reproduce
   
   See above as well as the attached `pyproject.toml`. Unfortunately, I cannot 
provide my dag.
   
   ### Operating System
   
   macOS 14.7.5 (23H527)
   
   ### Versions of Apache Airflow Providers
   
   These are the deps of the actual environment the dag will eventually need to 
run on (I have installed via `uv`):
   ```
   ❯ uv tree --depth 1
   Resolved 375 packages in 0.69ms
   local-dw-airflow v1.0.0a0
   ├── airflow-mcd v0.1.2
   ├── amplitude-analytics v1.1.1
   ├── apache-airflow v2.8.1
   ├── apache-airflow-providers-amazon v3.0.0
   ├── apache-airflow-providers-apache-cassandra v3.6.0
   ├── apache-airflow-providers-apache-hdfs v3.2.1
   ├── apache-airflow-providers-apache-hive v6.0.0
   ├── apache-airflow-providers-apache-spark v4.0.1
   ├── apache-airflow-providers-apache-sqoop v3.1.1
   ├── apache-airflow-providers-celery v3.6.0
   ├── apache-airflow-providers-cncf-kubernetes v8.0.0
   ├── apache-airflow-providers-common-sql v1.3.4
   ├── apache-airflow-providers-docker v3.5.1
   ├── apache-airflow-providers-elasticsearch v4.4.0
   ├── apache-airflow-providers-ftp v3.3.1
   ├── apache-airflow-providers-google v8.11.0
   ├── apache-airflow-providers-grpc v3.1.0
   ├── apache-airflow-providers-hashicorp v3.3.0
   ├── apache-airflow-providers-http v4.2.0
   ├── apache-airflow-providers-imap v3.1.1
   ├── apache-airflow-providers-microsoft-azure v5.2.1
   ├── apache-airflow-providers-mongo v4.2.0
   ├── apache-airflow-providers-mysql v4.0.2
   ├── apache-airflow-providers-odbc v3.2.1
   ├── apache-airflow-providers-postgres v5.4.0
   ├── apache-airflow-providers-presto v5.0.0
   ├── apache-airflow-providers-redis v3.1.0
   ├── apache-airflow-providers-sendgrid v3.1.0
   ├── apache-airflow-providers-sftp v4.2.4
   ├── apache-airflow-providers-slack v7.2.0
   ├── apache-airflow-providers-snowflake v4.0.4
   ├── apache-airflow-providers-sqlite v3.3.1
   ├── apache-airflow-providers-ssh v3.5.0
   ├── apache-airflow-providers-tableau v4.5.2
   ├── cassandra-driver v3.29.2
   ├── defusedxml v0.7.1
   ├── flask-session v0.5.0
   ├── mysql-connector-python v9.2.0
   ├── numpy v1.26.4
   ├── pycarlo v0.7.5
   ├── pykerberos v1.2.4
   ├── pymongo v4.9
   ├── pymysql v1.1.1
   ├── ruamel-yaml v0.17.21
   ├── ruamel-yaml-clib v0.2.6
   ├── salesforce-bulk v2.2.0
   ├── tableauserverclient v0.31
   ├── connexion[swagger-ui] v2.14.2 (group: dev)
   ├── graphviz v0.20.3 (group: dev)
   └── pyarrow v18.1.0 (group: dev)
   ```
   The issue still occurs when I cut it down to only have `apache-airflow` in 
the env. My dag itself does not have any non-airflow deps (it uses pendulum and 
httpx, which come with airflow).
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   This is the `pyproject.toml`:
   ```toml
   
####################################################################################################
   # Project
   
####################################################################################################
   
   [project]
   name = "local_dw_airflow"
   version = "1.0.0a0"
   description = "Local Airflow setup to test dags."
   requires-python = "==3.10.16"
   readme = "README.md"
   dependencies = [
       "airflow-mcd==0.1.2",
       "amplitude-analytics==1.1.1",
       "apache-airflow==2.8.1",
       "apache-airflow-providers-amazon==3.0.0",
       "apache-airflow-providers-apache-cassandra==3.6.0",
       "apache-airflow-providers-apache-hdfs==3.2.1",
       "apache-airflow-providers-apache-hive==6.0.0",
       "apache-airflow-providers-apache-spark==4.0.1",
       "apache-airflow-providers-apache-sqoop==3.1.1",
       "apache-airflow-providers-celery==3.6.0",
       "apache-airflow-providers-cncf-kubernetes==8.0.0",
       "apache-airflow-providers-common-sql==1.3.4",
       "apache-airflow-providers-docker==3.5.1",
       "apache-airflow-providers-elasticsearch==4.4.0",
       "apache-airflow-providers-ftp==3.3.1",
       "apache-airflow-providers-google==8.11.0",
       "apache-airflow-providers-grpc==3.1.0",
       "apache-airflow-providers-hashicorp==3.3.0",
       "apache-airflow-providers-http==4.2.0",
       "apache-airflow-providers-imap==3.1.1",
       "apache-airflow-providers-microsoft-azure==5.2.1",
       "apache-airflow-providers-mongo==4.2.0",
       "apache-airflow-providers-mysql==4.0.2",
       "apache-airflow-providers-odbc==3.2.1",
       "apache-airflow-providers-postgres==5.4.0",
       "apache-airflow-providers-presto==5.0.0",
       "apache-airflow-providers-redis==3.1.0",
       "apache-airflow-providers-sendgrid==3.1.0",
       "apache-airflow-providers-sftp==4.2.4",
       "apache-airflow-providers-slack==7.2.0",
       "apache-airflow-providers-snowflake==4.0.4",
       "apache-airflow-providers-sqlite==3.3.1",
       "apache-airflow-providers-ssh==3.5.0",
       "apache-airflow-providers-tableau==4.5.2",
       "cassandra-driver==3.29.2",
       "defusedxml==0.7.1",
       # NOTE: Versions after 0.5.0 don't have flask_session.sessions module 
which breaks other deps.
       "flask-session==0.5.0",
       "mysql-connector-python>=9.2.0",
       # NOTE: Numpy >=2.0.0 has binary incompat with pandas.
       "numpy==1.26.4",
       "pycarlo==0.7.5",
       "pykerberos==1.2.4",
       "pymongo==4.9",
       "pymysql>=1.1.1",
       "ruamel-yaml==0.17.21",
       "ruamel-yaml-clib==0.2.6",
       "salesforce-bulk>=2.2.0",
       "tableauserverclient==0.31",
   ]
   
   [dependency-groups]
   dev = ["connexion[swagger-ui]>=2.14.2", "graphviz>=0.20.3", "pyarrow<19.0.0"]
   
   
####################################################################################################
   # Ruff
   
####################################################################################################
   
   [tool.ruff]
   line-length = 100
   target-version = "py310"
   
   [tool.ruff.lint]
   ignore = [
       "B008", # do not perform function calls in argument defaults
       "C901", # too complex
       "E501", # line too long, handled by black
   ]
   preview = true
   select = [
       "B",    # flake8-bugbear
       "C4",   # flake8-comprehensions
       "D",    # pydocstyle
       "DOC",  # pydoclint
       "E",    # pycodestyle errors
       "F",    # pyflakes
       "FAST", # fastapi
       "FLY",  # flynt
       "FURB", # refurb
       "I",    # isort
       "N",    # pep8-naming
       "PERF", # perflint
       "RUF",  # ruff
       "UP",   # pyupgrade
       "W",    # pycodestyle warnings
   ]
   
   [tool.ruff.lint.pydocstyle]
   convention = "numpy"
   
   [tool.ruff.lint.isort]
   force-single-line = true
   lines-after-imports = 2
   
   ```
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to