Re: [DISCUSS] Using mprocs for standalone Airflow ?
If there's no one already taking it up, I can take a crack at it. -- Regards, Aritra Basu On Fri, 23 Jan 2026, 4:48 pm Jarek Potiuk, wrote: > I have created https://github.com/apache/airflow/issues/60971 to add > `--interactive` flag to airflow standalone using mprocs. It's not a "first" > issue - more of a small mini project, but not a difficult one for > someone who knows process handling and such. > > On Mon, Dec 29, 2025 at 8:53 AM Rahul Vats wrote: > > > +1 on this. The separate log panels would be a big help for debugging > > issues in standalone mode. > > > > Regards, > > Rahul Vats > > > > On Mon, 29 Dec 2025 at 11:25, Amogh Desai wrote: > > > > > > mprocs also publishes statically linked binaries > > > > > > Nice! > > > > > > I think I would best favour building it ourselves or publishing it and > > > using it > > > for the `--interactive` mode because these are the two best options > long > > > term. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Wed, Dec 24, 2025 at 1:36 PM Jarek Potiuk wrote: > > > > > > > > inherent dependency or we launch it using subprocess or we even > > > maintain > > > > a > > > > > minimalistic wrapper > > > > > around mprocs without the npm dependency. > > > > > My thought immediately goes to golang's embedded binaries > > > > > > > > Actually - (🤦) it's not an NPM app. I looked at the code. It's a > > > > (surprise, surprise) RUST app that has an NPM wrapper for easier > > > > installation only :D - that's what made me think it's npm.. That > would > > > > explain why it's so fast and nice :) > > > > > > > > So that is super easy. We don't need to "declare" mprocs as > dependency > > - > > > > when you look here: https://github.com/pvolok/mprocs/releases - > mprocs > > > > also > > > > publishes statically linked binaries which does not **require** npm > to > > be > > > > installed: > > > > > > > > So all we need to do when `--interactive` flag is used is to download > > the > > > > binary for the right architecture, decompress it to temporary > location > > > and > > > > run it from there (and we can cache it for later execution)., We > could > > > even > > > > copy and publish those binaries some place we control - somewhere in > > > > airflow.apache.org is good for that, similarly as we do with our > > chart - > > > > so > > > > that we do not have problems with GitHub rate limiting (and our > website > > > is > > > > already behind fastly cache, so it will be very fast to download it > by > > > > anyone). We could even - for maximum security, build those binaries > > > > ourselves from sources published by mprocs - > > > > https://github.com/pvolok/mprocs/tree/master/scripts has all the > > scripts > > > > needed to build those. > > > > > > > > J. > > > > > > > > > > > > > > > > On Wed, Dec 24, 2025 at 6:54 AM Amogh Desai > > > wrote: > > > > > > > > > Hi Jarek, > > > > > > > > > > I think it is a good idea to try and integrate mprocs with > standalone > > > > > airflow. > > > > > > > > > > While we are at it, we should try to see if we can somehow wrap > > around > > > > the > > > > > dependency mprocs > > > > > needed in a way that users working with standalone airflow do not > > > suffer > > > > at > > > > > all. It can maybe be an > > > > > inherent dependency or we launch it using subprocess or we even > > > maintain > > > > a > > > > > minimalistic wrapper > > > > > around mprocs without the npm dependency. > > > > > > > > > > My thought immediately goes to golang's embedded binaries which we > > > could > > > > > probably leverage to > > > > > design our own using wrapper or similar, maybe check out: > > > > > > > > > > > > > > > > > > > > https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c > > > > > > > > > > All in all, I think it's a good idea if it doesn't make user lives > > > harder > > > > > than it already is! > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Wed, Dec 24, 2025 at 3:11 AM Natanel > > > wrote: > > > > > > > > > > > Hello Jarek, I think that it is a great idea, it does look better > > and > > > > > seems > > > > > > to be more intuitive to use, when you have 1 screen at a time and > > you > > > > may > > > > > > change between them quickly and with ease, from experience, it > > would > > > > > > significantly improve the ease of use of a standalone airflow > > > instance. > > > > > > > > > > > > I think it will work well, and make it simpler to run airflow, > > > though, > > > > I > > > > > > think that Tmux is flexible enough to be able to reach a similar > > > > looking > > > > > UI > > > > > > with simple and intuitive controls, though it might require a lot > > of > > > > > trial > > > > > > and error, with configuration and community input in order to > > reach a > > > > > > simple and intuitive experience for a standalone airflow > instance. > > > > > > > > > > > > We might want to explore more solutions such as a more > > comprehen
Re: [DISCUSS] Using mprocs for standalone Airflow ?
I have created https://github.com/apache/airflow/issues/60971 to add `--interactive` flag to airflow standalone using mprocs. It's not a "first" issue - more of a small mini project, but not a difficult one for someone who knows process handling and such. On Mon, Dec 29, 2025 at 8:53 AM Rahul Vats wrote: > +1 on this. The separate log panels would be a big help for debugging > issues in standalone mode. > > Regards, > Rahul Vats > > On Mon, 29 Dec 2025 at 11:25, Amogh Desai wrote: > > > > mprocs also publishes statically linked binaries > > > > Nice! > > > > I think I would best favour building it ourselves or publishing it and > > using it > > for the `--interactive` mode because these are the two best options long > > term. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Wed, Dec 24, 2025 at 1:36 PM Jarek Potiuk wrote: > > > > > > inherent dependency or we launch it using subprocess or we even > > maintain > > > a > > > > minimalistic wrapper > > > > around mprocs without the npm dependency. > > > > My thought immediately goes to golang's embedded binaries > > > > > > Actually - (🤦) it's not an NPM app. I looked at the code. It's a > > > (surprise, surprise) RUST app that has an NPM wrapper for easier > > > installation only :D - that's what made me think it's npm.. That would > > > explain why it's so fast and nice :) > > > > > > So that is super easy. We don't need to "declare" mprocs as dependency > - > > > when you look here: https://github.com/pvolok/mprocs/releases - mprocs > > > also > > > publishes statically linked binaries which does not **require** npm to > be > > > installed: > > > > > > So all we need to do when `--interactive` flag is used is to download > the > > > binary for the right architecture, decompress it to temporary location > > and > > > run it from there (and we can cache it for later execution)., We could > > even > > > copy and publish those binaries some place we control - somewhere in > > > airflow.apache.org is good for that, similarly as we do with our > chart - > > > so > > > that we do not have problems with GitHub rate limiting (and our website > > is > > > already behind fastly cache, so it will be very fast to download it by > > > anyone). We could even - for maximum security, build those binaries > > > ourselves from sources published by mprocs - > > > https://github.com/pvolok/mprocs/tree/master/scripts has all the > scripts > > > needed to build those. > > > > > > J. > > > > > > > > > > > > On Wed, Dec 24, 2025 at 6:54 AM Amogh Desai > > wrote: > > > > > > > Hi Jarek, > > > > > > > > I think it is a good idea to try and integrate mprocs with standalone > > > > airflow. > > > > > > > > While we are at it, we should try to see if we can somehow wrap > around > > > the > > > > dependency mprocs > > > > needed in a way that users working with standalone airflow do not > > suffer > > > at > > > > all. It can maybe be an > > > > inherent dependency or we launch it using subprocess or we even > > maintain > > > a > > > > minimalistic wrapper > > > > around mprocs without the npm dependency. > > > > > > > > My thought immediately goes to golang's embedded binaries which we > > could > > > > probably leverage to > > > > design our own using wrapper or similar, maybe check out: > > > > > > > > > > > > > > https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c > > > > > > > > All in all, I think it's a good idea if it doesn't make user lives > > harder > > > > than it already is! > > > > > > > > Thanks & Regards, > > > > Amogh Desai > > > > > > > > > > > > On Wed, Dec 24, 2025 at 3:11 AM Natanel > > wrote: > > > > > > > > > Hello Jarek, I think that it is a great idea, it does look better > and > > > > seems > > > > > to be more intuitive to use, when you have 1 screen at a time and > you > > > may > > > > > change between them quickly and with ease, from experience, it > would > > > > > significantly improve the ease of use of a standalone airflow > > instance. > > > > > > > > > > I think it will work well, and make it simpler to run airflow, > > though, > > > I > > > > > think that Tmux is flexible enough to be able to reach a similar > > > looking > > > > UI > > > > > with simple and intuitive controls, though it might require a lot > of > > > > trial > > > > > and error, with configuration and community input in order to > reach a > > > > > simple and intuitive experience for a standalone airflow instance. > > > > > > > > > > We might want to explore more solutions such as a more > comprehensive > > > Tmux > > > > > configuration or maybe even Zellij, which is pretty similar to > Tmux, > > > and > > > > > are both lightweight, and simple to install (though, as Jen said, > it > > > will > > > > > make running airflow standalone a little harder). > > > > > > > > > > > I'd say it sounds like a good idea to have it as part of > standalone > > > > > airflow. One drawback it has that it has a dependency on npm, but > > also > > > it > > >
Re: [DISCUSS] Using mprocs for standalone Airflow ?
+1 on this. The separate log panels would be a big help for debugging issues in standalone mode. Regards, Rahul Vats On Mon, 29 Dec 2025 at 11:25, Amogh Desai wrote: > > mprocs also publishes statically linked binaries > > Nice! > > I think I would best favour building it ourselves or publishing it and > using it > for the `--interactive` mode because these are the two best options long > term. > > Thanks & Regards, > Amogh Desai > > > On Wed, Dec 24, 2025 at 1:36 PM Jarek Potiuk wrote: > > > > inherent dependency or we launch it using subprocess or we even > maintain > > a > > > minimalistic wrapper > > > around mprocs without the npm dependency. > > > My thought immediately goes to golang's embedded binaries > > > > Actually - (🤦) it's not an NPM app. I looked at the code. It's a > > (surprise, surprise) RUST app that has an NPM wrapper for easier > > installation only :D - that's what made me think it's npm.. That would > > explain why it's so fast and nice :) > > > > So that is super easy. We don't need to "declare" mprocs as dependency - > > when you look here: https://github.com/pvolok/mprocs/releases - mprocs > > also > > publishes statically linked binaries which does not **require** npm to be > > installed: > > > > So all we need to do when `--interactive` flag is used is to download the > > binary for the right architecture, decompress it to temporary location > and > > run it from there (and we can cache it for later execution)., We could > even > > copy and publish those binaries some place we control - somewhere in > > airflow.apache.org is good for that, similarly as we do with our chart - > > so > > that we do not have problems with GitHub rate limiting (and our website > is > > already behind fastly cache, so it will be very fast to download it by > > anyone). We could even - for maximum security, build those binaries > > ourselves from sources published by mprocs - > > https://github.com/pvolok/mprocs/tree/master/scripts has all the scripts > > needed to build those. > > > > J. > > > > > > > > On Wed, Dec 24, 2025 at 6:54 AM Amogh Desai > wrote: > > > > > Hi Jarek, > > > > > > I think it is a good idea to try and integrate mprocs with standalone > > > airflow. > > > > > > While we are at it, we should try to see if we can somehow wrap around > > the > > > dependency mprocs > > > needed in a way that users working with standalone airflow do not > suffer > > at > > > all. It can maybe be an > > > inherent dependency or we launch it using subprocess or we even > maintain > > a > > > minimalistic wrapper > > > around mprocs without the npm dependency. > > > > > > My thought immediately goes to golang's embedded binaries which we > could > > > probably leverage to > > > design our own using wrapper or similar, maybe check out: > > > > > > > > > https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c > > > > > > All in all, I think it's a good idea if it doesn't make user lives > harder > > > than it already is! > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Wed, Dec 24, 2025 at 3:11 AM Natanel > wrote: > > > > > > > Hello Jarek, I think that it is a great idea, it does look better and > > > seems > > > > to be more intuitive to use, when you have 1 screen at a time and you > > may > > > > change between them quickly and with ease, from experience, it would > > > > significantly improve the ease of use of a standalone airflow > instance. > > > > > > > > I think it will work well, and make it simpler to run airflow, > though, > > I > > > > think that Tmux is flexible enough to be able to reach a similar > > looking > > > UI > > > > with simple and intuitive controls, though it might require a lot of > > > trial > > > > and error, with configuration and community input in order to reach a > > > > simple and intuitive experience for a standalone airflow instance. > > > > > > > > We might want to explore more solutions such as a more comprehensive > > Tmux > > > > configuration or maybe even Zellij, which is pretty similar to Tmux, > > and > > > > are both lightweight, and simple to install (though, as Jen said, it > > will > > > > make running airflow standalone a little harder). > > > > > > > > > I'd say it sounds like a good idea to have it as part of standalone > > > > airflow. One drawback it has that it has a dependency on npm, but > also > > it > > > > has a standalone binary, that Airflow Standalone **could** download > > from > > > > https://github.com/pvolok/mprocs/releases and run internally. Or > maybe > > > > search for or even implement a simple version of it as an alternative > > in > > > > Python. > > > > > > > > I agree that using the npm version is not ideal, as it will add a > > > > dependency to airflow. > > > > > > > > A minimal version in python could also suffice, though it would take > > more > > > > time and add more code to maintain, though it would be the lightest > > > weight > > > > option, and using the binary install i
Re: [DISCUSS] Using mprocs for standalone Airflow ?
> mprocs also publishes statically linked binaries Nice! I think I would best favour building it ourselves or publishing it and using it for the `--interactive` mode because these are the two best options long term. Thanks & Regards, Amogh Desai On Wed, Dec 24, 2025 at 1:36 PM Jarek Potiuk wrote: > > inherent dependency or we launch it using subprocess or we even maintain > a > > minimalistic wrapper > > around mprocs without the npm dependency. > > My thought immediately goes to golang's embedded binaries > > Actually - (🤦) it's not an NPM app. I looked at the code. It's a > (surprise, surprise) RUST app that has an NPM wrapper for easier > installation only :D - that's what made me think it's npm.. That would > explain why it's so fast and nice :) > > So that is super easy. We don't need to "declare" mprocs as dependency - > when you look here: https://github.com/pvolok/mprocs/releases - mprocs > also > publishes statically linked binaries which does not **require** npm to be > installed: > > So all we need to do when `--interactive` flag is used is to download the > binary for the right architecture, decompress it to temporary location and > run it from there (and we can cache it for later execution)., We could even > copy and publish those binaries some place we control - somewhere in > airflow.apache.org is good for that, similarly as we do with our chart - > so > that we do not have problems with GitHub rate limiting (and our website is > already behind fastly cache, so it will be very fast to download it by > anyone). We could even - for maximum security, build those binaries > ourselves from sources published by mprocs - > https://github.com/pvolok/mprocs/tree/master/scripts has all the scripts > needed to build those. > > J. > > > > On Wed, Dec 24, 2025 at 6:54 AM Amogh Desai wrote: > > > Hi Jarek, > > > > I think it is a good idea to try and integrate mprocs with standalone > > airflow. > > > > While we are at it, we should try to see if we can somehow wrap around > the > > dependency mprocs > > needed in a way that users working with standalone airflow do not suffer > at > > all. It can maybe be an > > inherent dependency or we launch it using subprocess or we even maintain > a > > minimalistic wrapper > > around mprocs without the npm dependency. > > > > My thought immediately goes to golang's embedded binaries which we could > > probably leverage to > > design our own using wrapper or similar, maybe check out: > > > > > https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c > > > > All in all, I think it's a good idea if it doesn't make user lives harder > > than it already is! > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Wed, Dec 24, 2025 at 3:11 AM Natanel wrote: > > > > > Hello Jarek, I think that it is a great idea, it does look better and > > seems > > > to be more intuitive to use, when you have 1 screen at a time and you > may > > > change between them quickly and with ease, from experience, it would > > > significantly improve the ease of use of a standalone airflow instance. > > > > > > I think it will work well, and make it simpler to run airflow, though, > I > > > think that Tmux is flexible enough to be able to reach a similar > looking > > UI > > > with simple and intuitive controls, though it might require a lot of > > trial > > > and error, with configuration and community input in order to reach a > > > simple and intuitive experience for a standalone airflow instance. > > > > > > We might want to explore more solutions such as a more comprehensive > Tmux > > > configuration or maybe even Zellij, which is pretty similar to Tmux, > and > > > are both lightweight, and simple to install (though, as Jen said, it > will > > > make running airflow standalone a little harder). > > > > > > > I'd say it sounds like a good idea to have it as part of standalone > > > airflow. One drawback it has that it has a dependency on npm, but also > it > > > has a standalone binary, that Airflow Standalone **could** download > from > > > https://github.com/pvolok/mprocs/releases and run internally. Or maybe > > > search for or even implement a simple version of it as an alternative > in > > > Python. > > > > > > I agree that using the npm version is not ideal, as it will add a > > > dependency to airflow. > > > > > > A minimal version in python could also suffice, though it would take > more > > > time and add more code to maintain, though it would be the lightest > > weight > > > option, and using the binary install is probably better, in order to > not > > > rely on npm, especially due to the last seen events on the npm > repository > > > where a lot of malware was pushed to relatively big packages. > > > > > > There are a few considerations here, of whether we should bundle a > > > multiplexer or something similar to ariflow, just like is done with > > airflow > > > breeze, another option is always to open multiple new terminal windows > or > > > tabs i
Re: [DISCUSS] Using mprocs for standalone Airflow ?
> inherent dependency or we launch it using subprocess or we even maintain a > minimalistic wrapper > around mprocs without the npm dependency. > My thought immediately goes to golang's embedded binaries Actually - (🤦) it's not an NPM app. I looked at the code. It's a (surprise, surprise) RUST app that has an NPM wrapper for easier installation only :D - that's what made me think it's npm.. That would explain why it's so fast and nice :) So that is super easy. We don't need to "declare" mprocs as dependency - when you look here: https://github.com/pvolok/mprocs/releases - mprocs also publishes statically linked binaries which does not **require** npm to be installed: So all we need to do when `--interactive` flag is used is to download the binary for the right architecture, decompress it to temporary location and run it from there (and we can cache it for later execution)., We could even copy and publish those binaries some place we control - somewhere in airflow.apache.org is good for that, similarly as we do with our chart - so that we do not have problems with GitHub rate limiting (and our website is already behind fastly cache, so it will be very fast to download it by anyone). We could even - for maximum security, build those binaries ourselves from sources published by mprocs - https://github.com/pvolok/mprocs/tree/master/scripts has all the scripts needed to build those. J. On Wed, Dec 24, 2025 at 6:54 AM Amogh Desai wrote: > Hi Jarek, > > I think it is a good idea to try and integrate mprocs with standalone > airflow. > > While we are at it, we should try to see if we can somehow wrap around the > dependency mprocs > needed in a way that users working with standalone airflow do not suffer at > all. It can maybe be an > inherent dependency or we launch it using subprocess or we even maintain a > minimalistic wrapper > around mprocs without the npm dependency. > > My thought immediately goes to golang's embedded binaries which we could > probably leverage to > design our own using wrapper or similar, maybe check out: > > https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c > > All in all, I think it's a good idea if it doesn't make user lives harder > than it already is! > > Thanks & Regards, > Amogh Desai > > > On Wed, Dec 24, 2025 at 3:11 AM Natanel wrote: > > > Hello Jarek, I think that it is a great idea, it does look better and > seems > > to be more intuitive to use, when you have 1 screen at a time and you may > > change between them quickly and with ease, from experience, it would > > significantly improve the ease of use of a standalone airflow instance. > > > > I think it will work well, and make it simpler to run airflow, though, I > > think that Tmux is flexible enough to be able to reach a similar looking > UI > > with simple and intuitive controls, though it might require a lot of > trial > > and error, with configuration and community input in order to reach a > > simple and intuitive experience for a standalone airflow instance. > > > > We might want to explore more solutions such as a more comprehensive Tmux > > configuration or maybe even Zellij, which is pretty similar to Tmux, and > > are both lightweight, and simple to install (though, as Jen said, it will > > make running airflow standalone a little harder). > > > > > I'd say it sounds like a good idea to have it as part of standalone > > airflow. One drawback it has that it has a dependency on npm, but also it > > has a standalone binary, that Airflow Standalone **could** download from > > https://github.com/pvolok/mprocs/releases and run internally. Or maybe > > search for or even implement a simple version of it as an alternative in > > Python. > > > > I agree that using the npm version is not ideal, as it will add a > > dependency to airflow. > > > > A minimal version in python could also suffice, though it would take more > > time and add more code to maintain, though it would be the lightest > weight > > option, and using the binary install is probably better, in order to not > > rely on npm, especially due to the last seen events on the npm repository > > where a lot of malware was pushed to relatively big packages. > > > > There are a few considerations here, of whether we should bundle a > > multiplexer or something similar to ariflow, just like is done with > airflow > > breeze, another option is always to open multiple new terminal windows or > > tabs in the terminal, however, it might be too complex to do so with > > different working environments and people using all kinds of different > > terminals (and fonts) (i.e having to support windows, macos, xterm, > kitty, > > alacritty, Konsole, st and more), mprocs seems like a very good > candidate, > > but having to bundle it with airflow, could make the package heavier (or > > the installation more complex), and having an alternative python package > > will keep the airflow installation as simple and lightweight for > standalone > > a
Re: [DISCUSS] Using mprocs for standalone Airflow ?
Hi Jarek, I think it is a good idea to try and integrate mprocs with standalone airflow. While we are at it, we should try to see if we can somehow wrap around the dependency mprocs needed in a way that users working with standalone airflow do not suffer at all. It can maybe be an inherent dependency or we launch it using subprocess or we even maintain a minimalistic wrapper around mprocs without the npm dependency. My thought immediately goes to golang's embedded binaries which we could probably leverage to design our own using wrapper or similar, maybe check out: https://medium.com/dtoebe/embed-other-binaries-in-golang-binary-3f613314884c All in all, I think it's a good idea if it doesn't make user lives harder than it already is! Thanks & Regards, Amogh Desai On Wed, Dec 24, 2025 at 3:11 AM Natanel wrote: > Hello Jarek, I think that it is a great idea, it does look better and seems > to be more intuitive to use, when you have 1 screen at a time and you may > change between them quickly and with ease, from experience, it would > significantly improve the ease of use of a standalone airflow instance. > > I think it will work well, and make it simpler to run airflow, though, I > think that Tmux is flexible enough to be able to reach a similar looking UI > with simple and intuitive controls, though it might require a lot of trial > and error, with configuration and community input in order to reach a > simple and intuitive experience for a standalone airflow instance. > > We might want to explore more solutions such as a more comprehensive Tmux > configuration or maybe even Zellij, which is pretty similar to Tmux, and > are both lightweight, and simple to install (though, as Jen said, it will > make running airflow standalone a little harder). > > > I'd say it sounds like a good idea to have it as part of standalone > airflow. One drawback it has that it has a dependency on npm, but also it > has a standalone binary, that Airflow Standalone **could** download from > https://github.com/pvolok/mprocs/releases and run internally. Or maybe > search for or even implement a simple version of it as an alternative in > Python. > > I agree that using the npm version is not ideal, as it will add a > dependency to airflow. > > A minimal version in python could also suffice, though it would take more > time and add more code to maintain, though it would be the lightest weight > option, and using the binary install is probably better, in order to not > rely on npm, especially due to the last seen events on the npm repository > where a lot of malware was pushed to relatively big packages. > > There are a few considerations here, of whether we should bundle a > multiplexer or something similar to ariflow, just like is done with airflow > breeze, another option is always to open multiple new terminal windows or > tabs in the terminal, however, it might be too complex to do so with > different working environments and people using all kinds of different > terminals (and fonts) (i.e having to support windows, macos, xterm, kitty, > alacritty, Konsole, st and more), mprocs seems like a very good candidate, > but having to bundle it with airflow, could make the package heavier (or > the installation more complex), and having an alternative python package > will keep the airflow installation as simple and lightweight for standalone > as possible, the next step should the community decide to implement it in > python, will be research on the api's of different terminals to check if it > is possible to have a simple implementation to cover most of the terminal > emulators people are going to use. > > > Over all, adding such a tool will improve the airflow standalone experience > and benefit a good chunk of airflow users (and contributors) (who may want > to check their dags quickly and locally), and so maybe it is worth it to > make the installation a little harder or require one additional > *optional* dependency > to improve the standalone experience. > > Such an addition would be awesome. > > On Tue, 23 Dec 2025 at 21:20, Dheeraj Turaga > wrote: > > > Hi Jarek! > > > > I like this idea. We use airflow standalone a lot and more often its hard > > to identify issues when the log is multiplexed onto the screen. Also new > > users who are unaware that airflow drops a .json with a temp account in > > airflow home often find themselves scrambling to quickly copy the auto > > generated credentials before it vanishes into history. > > > > I can see instances where you may want to restart individual services > (dag > > processor, etc) without having to kill the whole standalone run and > launch > > a new one. > > > > Having separate logs for each service and a way to manage each one > > separately would be amazing > > > > I also agree with Jens here as to making the first time standalone run > > working with the most minimum number of commands as possible (pip install > > apache-airflow && airflow standalone) > > > > Adding th
Re: [DISCUSS] Using mprocs for standalone Airflow ?
Hello Jarek, I think that it is a great idea, it does look better and seems to be more intuitive to use, when you have 1 screen at a time and you may change between them quickly and with ease, from experience, it would significantly improve the ease of use of a standalone airflow instance. I think it will work well, and make it simpler to run airflow, though, I think that Tmux is flexible enough to be able to reach a similar looking UI with simple and intuitive controls, though it might require a lot of trial and error, with configuration and community input in order to reach a simple and intuitive experience for a standalone airflow instance. We might want to explore more solutions such as a more comprehensive Tmux configuration or maybe even Zellij, which is pretty similar to Tmux, and are both lightweight, and simple to install (though, as Jen said, it will make running airflow standalone a little harder). > I'd say it sounds like a good idea to have it as part of standalone airflow. One drawback it has that it has a dependency on npm, but also it has a standalone binary, that Airflow Standalone **could** download from https://github.com/pvolok/mprocs/releases and run internally. Or maybe search for or even implement a simple version of it as an alternative in Python. I agree that using the npm version is not ideal, as it will add a dependency to airflow. A minimal version in python could also suffice, though it would take more time and add more code to maintain, though it would be the lightest weight option, and using the binary install is probably better, in order to not rely on npm, especially due to the last seen events on the npm repository where a lot of malware was pushed to relatively big packages. There are a few considerations here, of whether we should bundle a multiplexer or something similar to ariflow, just like is done with airflow breeze, another option is always to open multiple new terminal windows or tabs in the terminal, however, it might be too complex to do so with different working environments and people using all kinds of different terminals (and fonts) (i.e having to support windows, macos, xterm, kitty, alacritty, Konsole, st and more), mprocs seems like a very good candidate, but having to bundle it with airflow, could make the package heavier (or the installation more complex), and having an alternative python package will keep the airflow installation as simple and lightweight for standalone as possible, the next step should the community decide to implement it in python, will be research on the api's of different terminals to check if it is possible to have a simple implementation to cover most of the terminal emulators people are going to use. Over all, adding such a tool will improve the airflow standalone experience and benefit a good chunk of airflow users (and contributors) (who may want to check their dags quickly and locally), and so maybe it is worth it to make the installation a little harder or require one additional *optional* dependency to improve the standalone experience. Such an addition would be awesome. On Tue, 23 Dec 2025 at 21:20, Dheeraj Turaga wrote: > Hi Jarek! > > I like this idea. We use airflow standalone a lot and more often its hard > to identify issues when the log is multiplexed onto the screen. Also new > users who are unaware that airflow drops a .json with a temp account in > airflow home often find themselves scrambling to quickly copy the auto > generated credentials before it vanishes into history. > > I can see instances where you may want to restart individual services (dag > processor, etc) without having to kill the whole standalone run and launch > a new one. > > Having separate logs for each service and a way to manage each one > separately would be amazing > > I also agree with Jens here as to making the first time standalone run > working with the most minimum number of commands as possible (pip install > apache-airflow && airflow standalone) > > Adding these via additional options would be nice! > > Thanks > Dheeraj > > On Tue, Dec 23, 2025 at 2:42 PM Jarek Potiuk wrote: > > > All good points: > > > So in standaonle mode we would need to mound one "dummy process" > > panel giving the key "getting started" information with UI URL and > > admin user/password at least. > > > > Great idea > > > > > We should not make it harder for users for first-time user > > experience. Today after a `pip install apache-airflow` you are > > potentially good to go > > > > Yes. Likely we can leave standalone behaviour working as of today by > > default > > > > > We therefore might consider > > not naming it `--mprocs` but rather `--interactive` to describe the > > use case? > > > > I really like this flag idea. > > > > > On the neutral side though I am not sure if a first time user is > > "easier" with merged logs to see errors of if a first time user might be > > overwhelmed with too many panels to scroll in to
Re: [DISCUSS] Using mprocs for standalone Airflow ?
Hi Jarek! I like this idea. We use airflow standalone a lot and more often its hard to identify issues when the log is multiplexed onto the screen. Also new users who are unaware that airflow drops a .json with a temp account in airflow home often find themselves scrambling to quickly copy the auto generated credentials before it vanishes into history. I can see instances where you may want to restart individual services (dag processor, etc) without having to kill the whole standalone run and launch a new one. Having separate logs for each service and a way to manage each one separately would be amazing I also agree with Jens here as to making the first time standalone run working with the most minimum number of commands as possible (pip install apache-airflow && airflow standalone) Adding these via additional options would be nice! Thanks Dheeraj On Tue, Dec 23, 2025 at 2:42 PM Jarek Potiuk wrote: > All good points: > > So in standaonle mode we would need to mound one "dummy process" > panel giving the key "getting started" information with UI URL and > admin user/password at least. > > Great idea > > > We should not make it harder for users for first-time user > experience. Today after a `pip install apache-airflow` you are > potentially good to go > > Yes. Likely we can leave standalone behaviour working as of today by > default > > > We therefore might consider > not naming it `--mprocs` but rather `--interactive` to describe the > use case? > > I really like this flag idea. > > > On the neutral side though I am not sure if a first time user is > "easier" with merged logs to see errors of if a first time user might be > overwhelmed with too many panels to scroll in to find an error not > knowing which component maybe raising a stack trace at all. > > Yep. With `--interactive` flag as non-default this issue would also be > handled. >
Re: [DISCUSS] Using mprocs for standalone Airflow ?
All good points: > So in standaonle mode we would need to mound one "dummy process" panel giving the key "getting started" information with UI URL and admin user/password at least. Great idea > We should not make it harder for users for first-time user experience. Today after a `pip install apache-airflow` you are potentially good to go Yes. Likely we can leave standalone behaviour working as of today by default > We therefore might consider not naming it `--mprocs` but rather `--interactive` to describe the use case? I really like this flag idea. > On the neutral side though I am not sure if a first time user is "easier" with merged logs to see errors of if a first time user might be overwhelmed with too many panels to scroll in to find an error not knowing which component maybe raising a stack trace at all. Yep. With `--interactive` flag as non-default this issue would also be handled.
Re: [DISCUSS] Using mprocs for standalone Airflow ?
Hi Jarek, I like the idea very much to have a better usability for standalone. That might make it easier for standalone first-user-experience. Being able to restart individual processes and seeing components individual logs is for sure a great thing. Nevertheless though three things to consider in my view: * Today when running standalone in most cases the intro URL and dynamically created admin password is printed in stdout, if mprocs is launched later this information would be hidden in the console. So in standaonle mode we would need to mound one "dummy process" panel giving the key "getting started" information with UI URL and admin user/password at least. Maybe also adding a link to "first usage help". * We should not make it harder for users for first-time user experience. Today after a `pip install apache-airflow` you are potentially good to go, adding mprocs as an additional (even small) manual dependency makes it harder to get started. Would be cool to have a pymprocs :-D * There might be usages where you run standaonle non-interactive in a backend (single container for PoC) - whereas mprocs would not be a good fit. Not sure whether this should be "default" or the interactive case should be the default. We therefore might consider not naming it `--mprocs` but rather `--interactive` to describe the use case? On the neutral side though I am not sure if a first time user is "easier" with merged logs to see errors of if a first time user might be overwhelmed with too many panels to scroll in to find an error not knowing which component maybe raising a stack trace at all. Jens On 12/23/25 12:25, Jarek Potiuk wrote: Hello here, After trying mprocs for "start-airflow" I think mprocs would be a great candidate to use for standalone airflow. I think currently one of the problematic things for standalone airflow is the log multiplexing at the output and lack of control over the individual processes run as part of "standalone" airflow execution. I think `mprocs` solves many of those problems that we have with "airflow standalone" nicely. It provides a very nice, terminal interface to monitor and control multiple processes - very user friendly, supporting all modern terminals, copy&pasting, mouse control for monitoring of running processes, ability to restart individual processes and it has also a number of interesting features in the roadmap. But even today it offers a lot more than we have today in standalone airflow: * separate logs for each process with nice navigation * easy copy&paste * keyboard shortcuts and navigation to switch and move between panels * ability to individually restart processes when needed * ability to autorestart processes when they fail * ability to run scripts defined in package.json (it's an npm tool) I'd say it sounds like a good idea to have it as part of standalone airflow. One drawback it has that it has a dependency on npm, but also it has a standalone binary, that Airflow Standalone **could** download from https://github.com/pvolok/mprocs/releases and run internally. Or maybe search for or even implement simple version of it as alternative in Python. We could have an `--mprocs/-no-mprocs` flag when starting the standalone airflow (and decide on default setting, also we could give the user a choice to run mprocs by default when they run standalone and download the binary, I think in general it delivers a very good and very user-friendly (and not even expert-friendly like tmux) interface and "airflow standalone" would get an immediate usability boost with it. I think the TUI interfaces of those sort have shown it's power over last years (k9s being the flagship tool). I would love to hear your thoughts about it. J.
