giri-builds opened a new issue, #71721:
URL: https://github.com/apache/airflow/issues/71721
### Under which category would you file this issue?
Task SDK
### Apache Airflow version
3.2.0
### What happened and how to reproduce it?
## What happened
I ran the same DAG and task workload on Apache Airflow 2.9.2 and 3.2.0.
The task emitted exactly the same payload:
- 100,001 payload lines
- 37.9 bytes of useful message content per line
Despite identical logical output, Airflow 3.2.0 produced substantially larger
task logs and took significantly longer to complete.
## Per-line record comparison
The following lines contain the same logical payload message.
Airflow 2.9.2:
[2026-08-16T13:54:02.571+0000 +0000] MainProcess MainThread
{subprocess.py:93} INFO - Line No 100000: Airflow 2 Log Size Test
Airflow 3.2.0:
{"timestamp":"2026-08-16T17:43:20.929261Z","level":"info","event":"Line No
100000: Airflow 3 Log Size
Test","logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook","filename":"subprocess.py","lineno":99}
The Airflow 3.2.0 record repeats a fully qualified logger identity for every
line:
"logger":"airflow.task.hooks.airflow.providers.standard.hooks.subprocess.SubprocessHook"
The useful event message is unchanged, but the Airflow 3.2.0 representation
adds structured fields for timestamp, level, logger, filename, and line number,
as well as JSON syntax.
The Airflow 2.9.2 representation uses a compact source reference:
{subprocess.py:93}
The expanded logger identity is a significant source of repeated per-line
metadata. It contributes to both storage amplification and the per-line
processing required to serialize and write task log records.
We are not claiming that the logger field alone explains the complete
regression. It is one directly observable contributor to the broader
structured-record overhead.
## Reproduction
The task emits 100,001 deterministic progress messages. The DAG, task code,
payload, worker resources, executor, logging backend, and deployment settings
were kept constant between the two tests. Only the Airflow version changed.
The task is representative of high-volume logging workloads.
## Results
──────────────────────┬───────────────┐
│ Metric │ AF 2.9.2 │ AF
3.2.0 │ Delta │
──────────────────────┼───────────────┤
│ Task log file size │ 12,493,072 B (11.91 MiB) │ 23,671,803
B (22.58 MiB) │ 1.89x │
──────────────────────┼───────────────┤
│ Payload lines emitted │ 100,001 │ 100,001
│ identical │
──────────────────────┼───────────────┤
│ Avg bytes per payload line │ 124.89 B │ 235.89 B
│ +111.0 B/line │
──────────────────────┼───────────────┤
│ Useful message content per line │ 37.9 B │ 37.9 B
│ identical │
──────────────────────┼───────────────┤
│ Signal ratio (content ÷ line) │ 30.3% │ 16.1%
│ −14.2 pp │
──────────────────────┼───────────────┤
│ gzip -6 size │ 378,417 B │ 732,143 B
│ 1.94x │
──────────────────────┼───────────────┤
│ Payload emit window │ 16.33 s │ 42.53 s*
│ 2.60x │
──────────────────────┼───────────────┤
│ Sustained log throughput │ 6,123 lines/s │ 2,351
lines/s* │ −62% │
──────────────────────┼───────────────┤
│ Total task wall time │ 16.86 s │ 44.43 s*
│ 2.64x │
──────────────────────┼───────────────┤
│ Framework (non-payload) lines │ 21 │ 259
│ 12.3x │
──────────────────────┼───────────────┤
│ Framework (non-payload) bytes │ 3,930 B │ 82,551 B
│ 21.0x │
──────────────────────┴───────────────┘
## Impact
The Airflow 3.2.0 task log is approximately 89.5% larger in bytes and
remains 1.94x larger after gzip compression.
The task takes approximately 163.5% longer overall, and sustained logging
throughput falls by approximately 62%.
The impact affects more than remote storage:
- Increased local and remote log size.
- Increased upload and transfer volume.
- Higher CPU and serialization overhead.
- Slower access to completed task logs in the Airflow UI.
- Slower scrolling and log rendering for large logs.
- Potentially increased risk of incomplete or failed UI rendering.
### What you think should happen instead?
## Expected behavior
For identical task output, Airflow 3.x should not introduce a significant
runtime and storage regression solely because task logs are represented using
structured records.
At minimum, users should have a supported deployment-level option to choose a
compact persisted task-log representation without modifying operators or
DAGs.
## Requested behavior
Please consider providing an opt-in compact or legacy task-log mode that:
1. Preserves the Airflow 3.x task-process/supervisor protocol.
2. Converts structured events to compact plaintext before persistence/upload,
if JSON is required internally.
3. Preserves timestamps, levels, tracebacks, multiline messages, ordering,
and
task-attempt boundaries.
4. Works with worker pods and supported executors.
5. Remains compatible with the Airflow UI, API, local handlers, and remote
logging backends.
6. Does not require modifications to operators or provider code.
Possible configuration concepts:
[logging]
task_log_format = legacy
or:
[logging]
structured_task_logs = false
The exact setting name is not important; the required capability is an
explicit, documented way to control the persisted task-log representation.
## Related reports
- #50274: task log format appears to ignore configuration
- #53442: Airflow 3.x logging configuration behavior
- Discussion #53006: Task SDK logs sent as JSON to the supervisor
- #55173: large task logs become slower to scroll
- #58226: large logs not rendering in the UI
- #57896: large logs do not appear in Airflow 3.1.2
- #54717: Airflow log tail support
- #45516: unable to see logs in the web UI
### Operating System
Amazon Linux 2023, x86_64
### Deployment
Other
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow 3.2.0
apache-airflow-core 3.2.0
apache-airflow-providers-amazon 9.23.0
apache-airflow-providers-apache-hive 9.4.0
apache-airflow-providers-cncf-kubernetes 10.14.0
apache-airflow-providers-common-compat 1.14.1
apache-airflow-providers-common-io 1.7.1
apache-airflow-providers-common-sql 1.33.0
apache-airflow-providers-fab 3.5.0
apache-airflow-providers-git 0.2.4
apache-airflow-providers-http 6.0.0
apache-airflow-providers-postgres 6.6.1
apache-airflow-providers-smtp 2.4.3
apache-airflow-providers-standard 1.12.1
apache-airflow-task-sdk 1.2.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
