Here's a doc I created on using Airflow with Windows/WSL. You can ignore most of it, but towards the end I describe how I setup PyCharm to debug. https://www.astronomer.io/guides/airflow-wsl/
Andrew On Thu, Jul 18, 2019 at 9:33 AM Andrew Harmon <[email protected]> wrote: > I use PyCharm as well to drop in break points. I usually run "airflow > test" or "airflow render" in the script path of the debug config, then step > through my code in the IDE. > > Andrew > > On Wed, Jul 17, 2019 at 7:48 AM Jarek Potiuk <[email protected]> > wrote: > >> Ah cool Ash. I Did not know about the 'render' command :). >> >> J. >> >> On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor <[email protected]> >> wrote: >> >> > And as for ways you can get the macro in a way you debug it - the >> `airflow >> > render` subcommand would help here. >> > >> > The other option would be to create a unit test file something like what >> > I've put below. >> > >> > (This just tests the default `{{ ds }}` macro, but this could be >> extended >> > to test a custom macro) >> > >> > -ash >> > >> > __unittest__ >> > from datetime import datetime >> > from unittest import TestCase >> > >> > from airflow import DAG >> > from airflow.models import DagRun, TaskInstance >> > from airflow.utils.db import create_session >> > from airflow.utils.timezone import make_aware >> > from airflow.utils.state import State >> > from airflow.operators.bash_operator import BashOperator >> > >> > >> > class MacroTest(TestCase): >> > def setUp(self): >> > self.dag = DAG( >> > dag_id="unittest", >> > start_date=datetime(2019, 1, 1) >> > ) >> > >> > def tearDown(self): >> > with create_session() as session: >> > session.query(DagRun).filter(DagRun.dag_id == >> > self.dag.dag_id).delete() >> > session.query(TaskInstance).filter(TaskInstance.dag_id == >> > self.dag.dag_id).delete() >> > >> > def test_macro(self): >> > # Use BashOperator as it's got easy macros to render >> > op = BashOperator( >> > task_id="test", >> > dag=self.dag, >> > bash_command='true', >> > env={'KEY': '{{ ds }}'}, >> > ) >> > >> > with create_session() as session: >> > # Set up the object to get a full template context >> > dr = self.dag.create_dagrun( >> > run_id="test", >> > execution_date=make_aware(datetime(2019, 1, 1)), >> > state=State.SCHEDULED, >> > session=session, >> > ) >> > >> > ti = dr.get_task_instance(op.task_id) >> > ti.task = op >> > >> > ti.render_templates() >> > >> > self.assertEqual(op.env['KEY'], '2019-01-01') >> > >> > >> > > On 17 Jul 2019, at 11:07, Jarek Potiuk <[email protected]> >> wrote: >> > > >> > > The best way to debug is to use ipdb debugger or remote debugging >> feature >> > > from PyCharm/IntelliJ. >> > > >> > > * For the first one ipdb - there is a note in upcoming Docker CI image >> > > environment: >> > > >> > >> https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests >> > > . >> > > You can set the ipdb trace method in any place of the python code you >> > have >> > > and you should be able to debug from the console. >> > > >> > > * For remote debugging it is a bit more difficult and requires paid >> > version >> > > of PyCharm/IntelliJ - I have another PR that I am going to propose >> soon - >> > > the Breeze environment and you can find description on how to setup >> > remote >> > > debugging here: >> > > >> > >> https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide >> > > >> > > >> > > Additionally Pycharm/IntelliJ allow you to debug various templates >> > > (including Jinja templates). You can likely combine this with remote >> > > debugging but I have never used it this way): >> > > https://blog.jetbrains.com/pycharm/2017/06/template-debugging/ >> > > >> > > J. >> > > >> > > On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. < >> > > [email protected]> wrote: >> > > >> > >> Hi all, >> > >> >> > >> I've just been working on updating many of my macros to include >> logic to >> > >> handle holiday calendars. There was a small mistake in one of the >> macros >> > >> and I found it very difficult to debug, I'm not sure where a default >> > logger >> > >> in the plugins would log out to and there was no obvious way to run >> the >> > >> code outside of airflow or somewhere I could add a breakpoint. >> > >> >> > >> My issue is solved now, but does anyone have an advise on how to >> debug >> > >> Macros or Plugins for future cases? >> > >> >> > >> Thanks, >> > >> Damian >> > >> >> > >> >> > >> =============================================================================== >> > >> >> > >> Please access the attached hyperlink for an important electronic >> > >> communications disclaimer: >> > >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html >> > >> >> > >> =============================================================================== >> > >> >> > >> >> > > >> > > >> > > -- >> > > >> > > Jarek Potiuk >> > > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > > >> > > M: +48 660 796 129 <+48660796129> >> > > [image: Polidea] <https://www.polidea.com/> >> > >> > >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> > > > -- > Andrew Harmon > (202) 615-6433 > -- Andrew Harmon (202) 615-6433
