This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch v3-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v3-1-test by this push:
new 4a73383c33f [v3-1-test] Improve documentation and behaviour of mprocs
(#58718) (#58798)
4a73383c33f is described below
commit 4a73383c33f7971336cf21f50f1c4d2410cb2d4e
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Nov 28 03:07:55 2025 +0100
[v3-1-test] Improve documentation and behaviour of mprocs (#58718) (#58798)
The AI-generated documentation had some hallucinations and had a bit
of duplication and chattines. Also it was not clear you can also easily
run mprocs outside of breeze in regular terminal (you can !) and how to
do it easily.
This is all corrected here:
* README.md is removed (most of the content was in QUICK REFERENCE or
was just too chatty
* Hallucinated keybindings were corrected
* basic.yaml is now simply mprocs.yml so that you can use it directly
in the contributing-docs/mprocs folder without specifying config file
* we are using now uv to run generate_mprocs_config.py and it has
inline requirements specified - making it easily usable outside of
breeze environment
* removed replacement of local executor that is not needed any more
also in run_mprocs
* added some non-obvious but useful key bindings in the guide
* the generated mprocs.yaml is now created in /files when run with
start-airflow which makes it directly usable outside of Breeze.
* there are some caveats of mprocs working with iTerm2 on MacOS, this
change documents workaround/solution needed to work with mprocs until
better solutions are found
(cherry picked from commit d1bd5559d1194a9b7626b75730ce62c840e5734e)
Co-authored-by: Jarek Potiuk <[email protected]>
---
.../images/iterm2-enable-mouse-reporting.png | Bin 0 -> 153945 bytes
contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md | 53 +++++++--
contributing-docs/mprocs/README.md | 123 ---------------------
.../mprocs/{basic.yaml => mprocs.yaml} | 0
scripts/in_container/bin/generate_mprocs_config.py | 8 +-
scripts/in_container/bin/run_mprocs | 12 +-
6 files changed, 52 insertions(+), 144 deletions(-)
diff --git a/contributing-docs/images/iterm2-enable-mouse-reporting.png
b/contributing-docs/images/iterm2-enable-mouse-reporting.png
new file mode 100644
index 00000000000..a8e23f83d02
Binary files /dev/null and
b/contributing-docs/images/iterm2-enable-mouse-reporting.png differ
diff --git a/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md
b/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md
index 25041f185d2..f50549e1b2a 100644
--- a/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md
+++ b/contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md
@@ -25,6 +25,11 @@
breeze start-airflow --use-mprocs
```
+Breeze will start Airflow components using mprocs to manage multiple processes
in a single terminal window.
+It will generate dynamically the required configuration based on the selected
executor and options and use it,
+the generated configuration file is stored in a `files` folder inside the
container, so that you can also
+use it outside of Breeze if needed.
+
## Common Usage Patterns
| Command | Description
|
@@ -38,16 +43,15 @@ breeze start-airflow --use-mprocs
## mprocs Keyboard Shortcuts
-| Key | Action |
-|---------|------------------------------|
-| `↑↓` | Navigate between processes |
-| `Space` | Show/hide process output |
-| `r` | Restart selected process |
-| `k` | Kill selected process |
-| `s` | Start selected process |
-| `a` | Toggle showing all processes |
-| `q` | Quit mprocs |
-| `?` | Show help |
+| Key | Action |
+|------|----------------------------|
+| `↑↓` | Navigate between processes |
+| `r` | Restart selected process |
+| `x` | Stop selected process |
+| `s` | Start selected process |
+| `a` | Add new process |
+| `q` | Quit mprocs |
+| `?` | Show help |
## Components Managed
@@ -97,3 +101,32 @@ curl -L
"https://github.com/pvolok/mprocs/releases/download/v${MPROCS_VERSION}/m
| tar -xz -C /usr/local/bin/ mprocs
chmod +x /usr/local/bin/mprocs
```
+
+## Mac OS X and iTerm2
+
+There are some known issues with mprocs and iTerm2 (which is often used as
terminal by
+developers using MacOS) with default settings, but they have solutions and
workarounds:
+
+* Mouse clicks are not captured correctly by default
+
+**Solution:** you need to configure "Enable Mouse reporting":
+
+
+
+* Ctrl-a does not work correctly when in breeze Docker container (issue
https://github.com/pvolok/mprocs/issues/179)
+
+**Workaround**: You need to use `Cmd - <-` to change focus instead of Ctrl-a.
Also you can switch
+between process list and output with mouse click providing that you applied
the solution above.
+
+## Standalone execution
+
+You can run mprocs outside of Breeze for custom setups. Create a `mprocs.yaml`
configuration
+file defining your processes, then start mprocs:
+
+```bash
+mprocs -f mprocs.yaml
+```
+
+An example [mprocs.yaml](mprocs.yaml) file for Airflow components can be found
in this directory - it has
+default configurations for all major Airflow components, but you can customize
it as needed by copying it
+elsewhere and modifying the process definitions, uncommenting the commented or
adding new processes.
diff --git a/contributing-docs/mprocs/README.md
b/contributing-docs/mprocs/README.md
deleted file mode 100644
index 8c0f4821699..00000000000
--- a/contributing-docs/mprocs/README.md
+++ /dev/null
@@ -1,123 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
- -->
-
-# mprocs Configuration for Airflow
-
-This directory contains mprocs configuration files for running Airflow
components.
-
-## Overview
-
-mprocs is a modern alternative to tmux for managing multiple processes. It
provides:
-
-- Intuitive TUI with better visual feedback
-- Easy process management (start, stop, restart)
-- Mouse and keyboard navigation
-- Process status indicators
-- Better cross-platform support
-
-## Files
-
-- `basic.yaml` - Static mprocs configuration example
-- `generate_mprocs_config.py` - Script to generate dynamic mprocs
configuration (located in `scripts/in_container/bin/`)
-- `run_mprocs` - Entry point script for starting Airflow with mprocs (located
in `scripts/in_container/bin/`)
-
-## Usage
-
-### Using Breeze
-
-The easiest way to use mprocs with Airflow is through the Breeze command:
-
-```bash
-breeze start-airflow --use-mprocs
-```
-
-This will:
-
-1. Start the Breeze container
-2. Dynamically generate an mprocs configuration based on your environment
-3. Launch mprocs with all configured Airflow components
-
-### With Debug Support
-
-You can combine mprocs with debugging:
-
-```bash
-breeze start-airflow --use-mprocs --debug scheduler --debug triggerer
-```
-
-### Manual Usage
-
-Inside the Breeze container, you can manually generate and use mprocs
configuration:
-
-```bash
-# Generate configuration
-python3 /opt/airflow/scripts/in_container/bin/generate_mprocs_config.py
/tmp/airflow-mprocs.yaml
-
-# Start mprocs
-mprocs --config /tmp/airflow-mprocs.yaml
-```
-
-## Dynamic Configuration
-
-The `generate_mprocs_config.py` script reads environment variables to
determine which components to run:
-
-- `INTEGRATION_CELERY` - Enables Celery worker
-- `CELERY_FLOWER` - Enables Flower UI
-- `STANDALONE_DAG_PROCESSOR` - Enables standalone DAG processor
-- `AIRFLOW__CORE__EXECUTOR` - Determines if Edge worker should run
-- `USE_AIRFLOW_VERSION` - Determines if API server (3.x+) or webserver (2.x)
should run
-- `BREEZE_DEBUG_*` - Enables debug mode for specific components
-- `DEV_MODE` - Enables development mode features
-
-## Components
-
-The following Airflow components can be managed by mprocs:
-
-1. **scheduler** - Main Airflow scheduler
-2. **api_server** (Airflow 3.x+) or **webserver** (Airflow 2.x)
-3. **triggerer** - Handles deferred tasks
-4. **dag_processor** - Standalone Dag processor (optional)
-5. **celery_worker** - Celery worker (when using CeleryExecutor)
-6. **flower** - Celery Flower monitoring UI (optional)
-7. **edge_worker** - Edge worker (when using EdgeExecutor)
-
-## mprocs Controls
-
-When mprocs is running:
-
-- **Arrow keys** - Navigate between processes
-- **Space** - Show/hide process output
-- **r** - Restart selected process
-- **k** - Kill selected process
-- **s** - Start selected process
-- **a** - Toggle showing all processes
-- **q** - Quit mprocs
-- **?** - Show help
-
-## Requirements
-
-mprocs is pre-installed in the Breeze CI image starting from Airflow 3.x
development. No additional setup is required.
-
-For older images or custom setups, you can manually install mprocs by adding
it to your Dockerfile or by customizing your Breeze image. See the installation
section in the Dockerfile.ci for reference.
-
-## See Also
-
-- [Debugging Airflow
Components](../../contributing-docs/20_debugging_airflow_components.rst)
-- [Contributors Quick
Start](../../contributing-docs/03_contributors_quick_start.rst)
-- [mprocs Quick
Reference](../../contributing-docs/mprocs/MPROCS_QUICK_REFERENCE.md)
diff --git a/contributing-docs/mprocs/basic.yaml
b/contributing-docs/mprocs/mprocs.yaml
similarity index 100%
rename from contributing-docs/mprocs/basic.yaml
rename to contributing-docs/mprocs/mprocs.yaml
diff --git a/scripts/in_container/bin/generate_mprocs_config.py
b/scripts/in_container/bin/generate_mprocs_config.py
index 7be9cf26b43..f793d5461ae 100755
--- a/scripts/in_container/bin/generate_mprocs_config.py
+++ b/scripts/in_container/bin/generate_mprocs_config.py
@@ -15,7 +15,13 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+# /// script
+# requires-python = ">=3.10"
+# dependencies = [
+# "rich>=13.6.0",
+# "PyYAML>=6.0",
+# ]
+# ///
"""Generate mprocs configuration dynamically based on environment variables."""
from __future__ import annotations
diff --git a/scripts/in_container/bin/run_mprocs
b/scripts/in_container/bin/run_mprocs
index a60377a0ba6..778b393eb7a 100755
--- a/scripts/in_container/bin/run_mprocs
+++ b/scripts/in_container/bin/run_mprocs
@@ -15,19 +15,11 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# Use LocalExecutor if not set and if backend is not sqlite as it gives
-# better performance
-if [[ ${BACKEND} != "sqlite" ]]; then
- export AIRFLOW__CORE__EXECUTOR=${AIRFLOW__CORE__EXECUTOR:-LocalExecutor}
-fi
-
# Generate mprocs configuration dynamically
-MPROCS_CONFIG_PATH="/tmp/mprocs-airflow.yaml"
-python /opt/airflow/scripts/in_container/bin/generate_mprocs_config.py
"${MPROCS_CONFIG_PATH}"
+MPROCS_CONFIG_PATH="/files/mprocs.yaml"
-if [[ $? -ne 0 ]]; then
+if ! uv run "/opt/airflow/scripts/in_container/bin/generate_mprocs_config.py"
"${MPROCS_CONFIG_PATH}"; then
echo "Failed to generate mprocs configuration"
exit 1
fi
