Re: Grouping tests using pytest markers

2019-12-27 Thread Tomasz Urbaszek
The suggestion of using -k flag is really interesting. It will require a lot of changes but adding marks will require the same effort. However, I think that using a marker is more explicit and easier to spot. Regarding "slow test" marker, I did a quick calculation of tests execution times

Re: Grouping tests using pytest markers

2019-12-27 Thread Darren Weber
Consider all the options for filtering tests: - http://doc.pytest.org/en/latest/example/markers.html The `pytest -k` filters are very useful. Provide guidelines on how to name things so that `pytest -k` can be used to filter categories of tests. Use markers for tests that might be the

General-purpose SQL hook/operator

2019-12-27 Thread Bjorn Olsen
Hi community I have a question/suggestion about the different SQL Database hooks + operators. Examples: - MsSqlHook (Implements DbApiHook using pymssql) - MySqlHook (Implements DbApiHook using MySQLdb) - PostgresHook (Implements DbApiHook using psycopg2) These hooks tend to have

Re: [DISCUSS] Performance aspects of Airflow (2.0 + 1.10 backport)

2019-12-27 Thread Jarek Potiuk
> > > As I mentioned in that article I think it's critical both for performance > testing of changes, and "regression" testing of releases that we have a > repeatable performance test framework. I somewhat simplified the script I > used in the article, so I need to tidy it up a bit more and work

Re: Grouping tests using pytest markers

2019-12-27 Thread Tomasz Urbaszek
+1 for integrations and backends, it's a good start ;) T. On Fri, Dec 27, 2019 at 12:16 PM Jarek Potiuk wrote: > Since I am going to start working on it soon - I'd love to get some > opinions :). > > J. > > On Mon, Dec 23, 2019 at 11:13 AM Jarek Potiuk > wrote: > > > I have a concrete

Re: Grouping tests using pytest markers

2019-12-27 Thread Jarek Potiuk
Since I am going to start working on it soon - I'd love to get some opinions :). J. On Mon, Dec 23, 2019 at 11:13 AM Jarek Potiuk wrote: > I have a concrete proposal that we can start with. It's not a final set of > markers we might want to have but one that we can start with and make an >

Re: Call to organize more First time Airflow Contributor's Workshops

2019-12-27 Thread Jarek Potiuk
Yep. Tomek is driving the effort to start the Apache Local Community and I am helping with that :). Would be great to see more of those - starting around more focused communities - we organise already Apache Beam and Apache Airflow meetups so that's a good start :) Speaking of workshops - again

Re: Airflow Celery Executor and Postgres idle connection problem

2019-12-27 Thread Jarek Potiuk
You might have some code in your DAG to open and not close those connections. I think in normal circumstances there should be connection pools used and no connections should be left-over in idle state. Which version of Airflow are you using? Can you make a quick test and modify your DAGs to do