Hey Andrew, Fantastic! Would you mind if I use some of this in the docs for the upcoming Simplified Development Workflow - AIP-7 (look out for the webinar next week on Thursday) ? Maybe you would also like to help with testing of the new "Breeze" simplified development environment in Windows + WSL environment?
One of the aims of Breeze is to make a consistent environment for development and debugging (including IDE) but Windows was so far not in scope. I did a lot of testing for MacOS and some for Linux (with other's help) but Windows + WSL is a bit out of my reach for now. Also some of the upcoming improvements like pre-commit checks are Linux/MacOS tested for now. That would be awesome to also target Windows. So far we targeted mostly MacOS/Linux Developers - however now after AIP-10 pretty much all of the heavy lifting is done in Docker, and the only integration points you have with local development are bash scripts. SO maybe using WSL on Windows and having someone who can test it and use it daily, is a good idea to bring it as "officially supported" environment and have all Breeze features available there as well ? Some more information about the upcoming Breeze in AIP-7 https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow J. On Thu, Jul 18, 2019 at 3:36 PM Andrew Harmon <[email protected]> wrote: > Here's a doc I created on using Airflow with Windows/WSL. You can ignore > most of it, but towards the end I describe how I setup PyCharm to debug. > https://www.astronomer.io/guides/airflow-wsl/ > > Andrew > > On Thu, Jul 18, 2019 at 9:33 AM Andrew Harmon <[email protected]> > wrote: > > > I use PyCharm as well to drop in break points. I usually run "airflow > > test" or "airflow render" in the script path of the debug config, then > step > > through my code in the IDE. > > > > Andrew > > > > On Wed, Jul 17, 2019 at 7:48 AM Jarek Potiuk <[email protected]> > > wrote: > > > >> Ah cool Ash. I Did not know about the 'render' command :). > >> > >> J. > >> > >> On Wed, Jul 17, 2019 at 12:49 PM Ash Berlin-Taylor <[email protected]> > >> wrote: > >> > >> > And as for ways you can get the macro in a way you debug it - the > >> `airflow > >> > render` subcommand would help here. > >> > > >> > The other option would be to create a unit test file something like > what > >> > I've put below. > >> > > >> > (This just tests the default `{{ ds }}` macro, but this could be > >> extended > >> > to test a custom macro) > >> > > >> > -ash > >> > > >> > __unittest__ > >> > from datetime import datetime > >> > from unittest import TestCase > >> > > >> > from airflow import DAG > >> > from airflow.models import DagRun, TaskInstance > >> > from airflow.utils.db import create_session > >> > from airflow.utils.timezone import make_aware > >> > from airflow.utils.state import State > >> > from airflow.operators.bash_operator import BashOperator > >> > > >> > > >> > class MacroTest(TestCase): > >> > def setUp(self): > >> > self.dag = DAG( > >> > dag_id="unittest", > >> > start_date=datetime(2019, 1, 1) > >> > ) > >> > > >> > def tearDown(self): > >> > with create_session() as session: > >> > session.query(DagRun).filter(DagRun.dag_id == > >> > self.dag.dag_id).delete() > >> > session.query(TaskInstance).filter(TaskInstance.dag_id == > >> > self.dag.dag_id).delete() > >> > > >> > def test_macro(self): > >> > # Use BashOperator as it's got easy macros to render > >> > op = BashOperator( > >> > task_id="test", > >> > dag=self.dag, > >> > bash_command='true', > >> > env={'KEY': '{{ ds }}'}, > >> > ) > >> > > >> > with create_session() as session: > >> > # Set up the object to get a full template context > >> > dr = self.dag.create_dagrun( > >> > run_id="test", > >> > execution_date=make_aware(datetime(2019, 1, 1)), > >> > state=State.SCHEDULED, > >> > session=session, > >> > ) > >> > > >> > ti = dr.get_task_instance(op.task_id) > >> > ti.task = op > >> > > >> > ti.render_templates() > >> > > >> > self.assertEqual(op.env['KEY'], '2019-01-01') > >> > > >> > > >> > > On 17 Jul 2019, at 11:07, Jarek Potiuk <[email protected]> > >> wrote: > >> > > > >> > > The best way to debug is to use ipdb debugger or remote debugging > >> feature > >> > > from PyCharm/IntelliJ. > >> > > > >> > > * For the first one ipdb - there is a note in upcoming Docker CI > image > >> > > environment: > >> > > > >> > > >> > https://github.com/PolideaInternal/airflow/blob/ms-travis-ci-tests/CONTRIBUTING.md#running-individual-tests > >> > > . > >> > > You can set the ipdb trace method in any place of the python code > you > >> > have > >> > > and you should be able to debug from the console. > >> > > > >> > > * For remote debugging it is a bit more difficult and requires paid > >> > version > >> > > of PyCharm/IntelliJ - I have another PR that I am going to propose > >> soon - > >> > > the Breeze environment and you can find description on how to setup > >> > remote > >> > > debugging here: > >> > > > >> > > >> > https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst#debugging-airflow-breeze-tests-in-ide > >> > > > >> > > > >> > > Additionally Pycharm/IntelliJ allow you to debug various templates > >> > > (including Jinja templates). You can likely combine this with remote > >> > > debugging but I have never used it this way): > >> > > https://blog.jetbrains.com/pycharm/2017/06/template-debugging/ > >> > > > >> > > J. > >> > > > >> > > On Tue, Jul 16, 2019 at 9:23 PM Shaw, Damian P. < > >> > > [email protected]> wrote: > >> > > > >> > >> Hi all, > >> > >> > >> > >> I've just been working on updating many of my macros to include > >> logic to > >> > >> handle holiday calendars. There was a small mistake in one of the > >> macros > >> > >> and I found it very difficult to debug, I'm not sure where a > default > >> > logger > >> > >> in the plugins would log out to and there was no obvious way to run > >> the > >> > >> code outside of airflow or somewhere I could add a breakpoint. > >> > >> > >> > >> My issue is solved now, but does anyone have an advise on how to > >> debug > >> > >> Macros or Plugins for future cases? > >> > >> > >> > >> Thanks, > >> > >> Damian > >> > >> > >> > >> > >> > > >> > =============================================================================== > >> > >> > >> > >> Please access the attached hyperlink for an important electronic > >> > >> communications disclaimer: > >> > >> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > >> > >> > >> > > >> > =============================================================================== > >> > >> > >> > >> > >> > > > >> > > > >> > > -- > >> > > > >> > > Jarek Potiuk > >> > > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > > >> > > M: +48 660 796 129 <+48660796129> > >> > > [image: Polidea] <https://www.polidea.com/> > >> > > >> > > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > > > > > > -- > > Andrew Harmon > > (202) 615-6433 > > > > > -- > Andrew Harmon > (202) 615-6433 > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
