I use PyCharm as well to drop in break points. I usually run "airflow test" or "airflow render" in the script path of the debug config, then step through my code in the IDE.
Andrew On Wed, Jul 17, 2019 at 7:48 AM Jarek Potiuk <[email protected]> wrote: > Ah cool Ash. I Did not know about the 'render' command :). > > J. > > On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor <[email protected]> wrote: > > > And as for ways you can get the macro in a way you debug it - the > `airflow > > render` subcommand would help here. > > > > The other option would be to create a unit test file something like what > > I've put below. > > > > (This just tests the default `{{ ds }}` macro, but this could be extended > > to test a custom macro) > > > > -ash > > > > __unittest__ > > from datetime import datetime > > from unittest import TestCase > > > > from airflow import DAG > > from airflow.models import DagRun, TaskInstance > > from airflow.utils.db import create_session > > from airflow.utils.timezone import make_aware > > from airflow.utils.state import State > > from airflow.operators.bash_operator import BashOperator > > > > > > class MacroTest(TestCase): > > def setUp(self): > > self.dag = DAG( > > dag_id="unittest", > > start_date=datetime(2019, 1, 1) > > ) > > > > def tearDown(self): > > with create_session() as session: > > session.query(DagRun).filter(DagRun.dag_id == > > self.dag.dag_id).delete() > > session.query(TaskInstance).filter(TaskInstance.dag_id == > > self.dag.dag_id).delete() > > > > def test_macro(self): > > # Use BashOperator as it's got easy macros to render > > op = BashOperator( > > task_id="test", > > dag=self.dag, > > bash_command='true', > > env={'KEY': '{{ ds }}'}, > > ) > > > > with create_session() as session: > > # Set up the object to get a full template context > > dr = self.dag.create_dagrun( > > run_id="test", > > execution_date=make_aware(datetime(2019, 1, 1)), > > state=State.SCHEDULED, > > session=session, > > ) > > > > ti = dr.get_task_instance(op.task_id) > > ti.task = op > > > > ti.render_templates() > > > > self.assertEqual(op.env['KEY'], '2019-01-01') > > > > > > > On 17 Jul 2019, at 11:07, Jarek Potiuk <[email protected]> > wrote: > > > > > > The best way to debug is to use ipdb debugger or remote debugging > feature > > > from PyCharm/IntelliJ. > > > > > > * For the first one ipdb - there is a note in upcoming Docker CI image > > > environment: > > > > > > https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests > > > . > > > You can set the ipdb trace method in any place of the python code you > > have > > > and you should be able to debug from the console. > > > > > > * For remote debugging it is a bit more difficult and requires paid > > version > > > of PyCharm/IntelliJ - I have another PR that I am going to propose > soon - > > > the Breeze environment and you can find description on how to setup > > remote > > > debugging here: > > > > > > https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide > > > > > > > > > Additionally Pycharm/IntelliJ allow you to debug various templates > > > (including Jinja templates). You can likely combine this with remote > > > debugging but I have never used it this way): > > > https://blog.jetbrains.com/pycharm/2017/06/template-debugging/ > > > > > > J. > > > > > > On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. < > > > [email protected]> wrote: > > > > > >> Hi all, > > >> > > >> I've just been working on updating many of my macros to include logic > to > > >> handle holiday calendars. There was a small mistake in one of the > macros > > >> and I found it very difficult to debug, I'm not sure where a default > > logger > > >> in the plugins would log out to and there was no obvious way to run > the > > >> code outside of airflow or somewhere I could add a breakpoint. > > >> > > >> My issue is solved now, but does anyone have an advise on how to debug > > >> Macros or Plugins for future cases? > > >> > > >> Thanks, > > >> Damian > > >> > > >> > > > =============================================================================== > > >> > > >> Please access the attached hyperlink for an important electronic > > >> communications disclaimer: > > >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > >> > > > =============================================================================== > > >> > > >> > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Andrew Harmon (202) 615-6433
