It's not useful for online profiling, but to get a profile of specific things using the builtin cProfile module I've leveraged the airflow test CLI command, there's an example here: https://gist.github.com/jiffyclub/a3204998c8190abd8ea50090c30347fa The context here was profiling why a specific DAG was taking a long time to build.
On Wed, Jun 28, 2017 at 9:45 AM Bolke de Bruin <[email protected]> wrote: > Are you able to share some of the results/insights from this? Particularly > on Airflow’s internals of course. > > Bolke > > > On 27 Jun 2017, at 22:40, Alex Guziel <[email protected]> > wrote: > > > > Yeah, actually we have setup Newrelic for Airflow too at Airbnb, which > > gives decent insights into webserver perf. In terms of SQL queries, > adding > > `echo=True` to the SQLAlchemy engine creation is pretty good for seeing > > which sql queries get created. I tried some Python profilers before but > > they weren't super helpful. > > > > On Tue, Jun 27, 2017 at 1:27 PM, Maxime Beauchemin < > > [email protected]> wrote: > > > >> Nice. It would be great if DAG parsing was faster, and some of the > >> endpoints on the website have grown really slow as you we've grown the > >> number of DAGs, and on the DAGs with large number of tasks. > >> > >> I had the intuition that DAG parsing could be faster if operators > >> late-imported hooks (who themselves import external libs) but I have no > >> evidence or test to support it. > >> > >> I'm sure there's tons of low hanging fruit and this type of tool should > >> make it really clear. > >> > >> We've set up NewRelic (which seems similar as this tooling at first > sight) > >> for Superset at Airbnb and it gave us great insight. > >> > >> Max > >> > >> On Tue, Jun 27, 2017 at 1:01 PM, Bolke de Bruin <[email protected]> > wrote: > >> > >>> Free version also there, maybe more integration testing and > benchmarking. > >>> > >>> https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/> > >>> > >>> B. > >>> > >>>> On 27 Jun 2017, at 22:00, Chris Riccomini <[email protected]> > >> wrote: > >>>> > >>>> Seems you have to pay? > >>>> > >>>> On Tue, Jun 27, 2017 at 12:56 PM, Bolke de Bruin <[email protected]> > >>> wrote: > >>>> > >>>>> Just saw this tool on hacker news: > >>>>> > >>>>> https://github.com/stackimpact/stackimpact-python < > >> https://github.com/ > >>>>> stackimpact/stackimpact-python> > >>>>> > >>>>> Might be interesting for some profiling. > >>>>> > >>>>> Bolke > >>> > >>> > >> > >
