Are you able to share some of the results/insights from this? Particularly on Airflow’s internals of course.
Bolke > On 27 Jun 2017, at 22:40, Alex Guziel <[email protected]> wrote: > > Yeah, actually we have setup Newrelic for Airflow too at Airbnb, which > gives decent insights into webserver perf. In terms of SQL queries, adding > `echo=True` to the SQLAlchemy engine creation is pretty good for seeing > which sql queries get created. I tried some Python profilers before but > they weren't super helpful. > > On Tue, Jun 27, 2017 at 1:27 PM, Maxime Beauchemin < > [email protected]> wrote: > >> Nice. It would be great if DAG parsing was faster, and some of the >> endpoints on the website have grown really slow as you we've grown the >> number of DAGs, and on the DAGs with large number of tasks. >> >> I had the intuition that DAG parsing could be faster if operators >> late-imported hooks (who themselves import external libs) but I have no >> evidence or test to support it. >> >> I'm sure there's tons of low hanging fruit and this type of tool should >> make it really clear. >> >> We've set up NewRelic (which seems similar as this tooling at first sight) >> for Superset at Airbnb and it gave us great insight. >> >> Max >> >> On Tue, Jun 27, 2017 at 1:01 PM, Bolke de Bruin <[email protected]> wrote: >> >>> Free version also there, maybe more integration testing and benchmarking. >>> >>> https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/> >>> >>> B. >>> >>>> On 27 Jun 2017, at 22:00, Chris Riccomini <[email protected]> >> wrote: >>>> >>>> Seems you have to pay? >>>> >>>> On Tue, Jun 27, 2017 at 12:56 PM, Bolke de Bruin <[email protected]> >>> wrote: >>>> >>>>> Just saw this tool on hacker news: >>>>> >>>>> https://github.com/stackimpact/stackimpact-python < >> https://github.com/ >>>>> stackimpact/stackimpact-python> >>>>> >>>>> Might be interesting for some profiling. >>>>> >>>>> Bolke >>> >>> >>
