Yeah, actually we have setup Newrelic for Airflow too at Airbnb, which gives decent insights into webserver perf. In terms of SQL queries, adding `echo=True` to the SQLAlchemy engine creation is pretty good for seeing which sql queries get created. I tried some Python profilers before but they weren't super helpful.
On Tue, Jun 27, 2017 at 1:27 PM, Maxime Beauchemin < [email protected]> wrote: > Nice. It would be great if DAG parsing was faster, and some of the > endpoints on the website have grown really slow as you we've grown the > number of DAGs, and on the DAGs with large number of tasks. > > I had the intuition that DAG parsing could be faster if operators > late-imported hooks (who themselves import external libs) but I have no > evidence or test to support it. > > I'm sure there's tons of low hanging fruit and this type of tool should > make it really clear. > > We've set up NewRelic (which seems similar as this tooling at first sight) > for Superset at Airbnb and it gave us great insight. > > Max > > On Tue, Jun 27, 2017 at 1:01 PM, Bolke de Bruin <[email protected]> wrote: > > > Free version also there, maybe more integration testing and benchmarking. > > > > https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/> > > > > B. > > > > > On 27 Jun 2017, at 22:00, Chris Riccomini <[email protected]> > wrote: > > > > > > Seems you have to pay? > > > > > > On Tue, Jun 27, 2017 at 12:56 PM, Bolke de Bruin <[email protected]> > > wrote: > > > > > >> Just saw this tool on hacker news: > > >> > > >> https://github.com/stackimpact/stackimpact-python < > https://github.com/ > > >> stackimpact/stackimpact-python> > > >> > > >> Might be interesting for some profiling. > > >> > > >> Bolke > > > > >
