Airflow Releases

Airflow Developers, We were looking at writing a workflow framework in Python when we found Airflow. We have carried out some proof of concept work for using Airflow and wish to continue using it as it comes with lots of great features out-of-the-box. We have created our own fork here:

Skip task

Hi All, I am using Airflow 1.7.1.3 and have a particular requirement, which I don't think is currently supported by Airflow but just wanted to check in case I was missing something. I occasionally wish to skip a particular task in a given DAG run such that the task does not run for that DAG

Re: Skip task

some minutes or hours into the future before it's looked at again, unless you really dive into airflow, scheduling semantics (@once vs. other schedules) and how the scheduler works. Rgds, Gerard On Tue, Nov 8, 2016 at 5:30 PM, Maycock, Luke < luke.mayc...@affiliate.oliverwyman.com> wrote:

Re: Airflow Logging

wyman.com/> ____ From: Maycock, Luke Sent: 13 October 2016 14:52 To: dev@airflow.incubator.apache.org Subject: Airflow Logging Hi All, We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a high level design for how to improve the logging throughout the Airflow code to be mor

Re: Airflow Logging

e one of the worst module in the standard library and the cfg not-so-standard standard isn't great. We have many instances where we allow for people to pass objects or function as configuration parameters anyways, so let's keep the whole configuration in one place! Max On Fri, Oct 14, 2016 at 7:43

Python 3

Hi All, I have been using Airflow for a while now and have a couple of questions that I am hoping someone knows the answer to: 1. Is Airflow now Python 3 compatible? The documentation used to state that Airflow was only compatible with Python 2.7

Airflow Logging

Hi All, We (owlabs - fork: https://github.com/owlabs/incubator-airflow) have a high level design for how to improve the logging throughout the Airflow code to be more consistent, maintainable and extensible. We'd really appreciate any feedback on the design. Design for Consolidating Logging

Re: scheduler questions

This is great! I have also upvoted https://github.com/apache/incubator-airflow/pull/1830. If I would like to get feedback on a design or if I have an Airflow question, is emailing this dev list the correct way to go? Cheers, Luke Maycock OLIVER WYMAN

Re: String formatting

ator.apache.org Cc: Maycock, Luke Subject: Re: String formatting Indeed -- though I think the larger question from Luke is whether or not we want to enforce a certain style of logging message (variable arguments vs formatting the string itself). Since there's nothing to stop users from formatting the

String formatting

Hi Dev List, We're currently working on removing all of the new Landscape.io warnings from some of our code and we're noticing the following quite a lot: "Use % formatting in logging functions and pass the % parameters as arguments" This is being flagged up on lines such as:

Re: How to abort a DagRun?

Hi Christian, I believe you can achieve this by going to 'Browse > Dag Runs'. Find the DAG run you wish to stop and click the check box on the left. Next, click 'With selected' and set it to either 'success' or 'failed'. I believe this will prevent any further tasks in the DAG run from

Re: Skip task

che/incubator-airflow/blob/master/airflow/migrations/versions/4addfa1236f1_add_fractional_seconds_to_mysql_tables.py> Make sure to include "type_=mysql.DATETIME(fsp=6)” for your DateTime types on MySQL. - Bolke > Op 12 dec. 2016, om 11:33 heeft Maycock, Luke > <luke.mayc...@a

Re: Skip task

beyond). I might have missed something. Thanks Bolke Sent from my iPhone > On 9 Dec 2016, at 14:27, Maycock, Luke > <luke.mayc...@affiliate.oliverwyman.com> wrote: > > I found the issue to be that, for MySQL, the datetime was being rounded to > the nearest second. The stra

Re: Travis build failure

I have only noticed this. All recent PRs have this error. At a glance, I could not see any individual commit that caused this so maybe this is an environment issue. I have not really spent any time looking into the issue yet but if I find a solution, I will let you know. Cheers, Luke Maycock

API additions

Hello, We've had a requirement for Airflow to be able to start DAG Runs and check on the state of tasks in a specific DAG Run via API. We'd seen that an experimental API had been put in place and expanded on this, but there has since been a significant overhaul, mostly around security and the

Re: Travis build failure

nary seems to missing from the path for some reason. I’ll > have > >> a look to see if I can fix that. > >> > >>> Op 5 dec. 2016, om 14:20 heeft Maycock, Luke <luke.maycock@affiliate. > >> oliverwyman.com> het volgende geschreven: > >>> &g

Re: Python 3

www.conornash.com | @conornash <https://twitter.com/conornash> On Sun, Jan 1, 2017 at 9:12 PM, George Leslie-Waksman < geo...@cloverhealth.com.invalid> wrote: > We are using Airflow with Python3, PostgreSQL and Redis; it works just > fine. > > On Thu, Dec 1, 2016 at 10: