Re: [VOTE] Airflow 1.9.0rc4

I'm fairly sure I'm back to stock config now. Here is the DAG I am testing with: from datetime import datetime from airflow import DAG from airflow.operators.python_operator import PythonOperator dag_args = { "start_date": datetime(2017, 11, 20), } dag = DAG("tests", default_args=dag_args)

Re: Enabling flake8

Yes. It is now enforced (travis will fail in case thr linting has not been done). Bolke Op 30 nov. 2017 5:03 p.m. schreef "Andy Hadjigeorgiou" : > This is great! I've been following the guidelines here >

Re: RBAC Update

Given I have merge rights on FAB I could probably do another round of review and get your PRs through. I would really like to get the main maintainer's input on things that touch the core (composite-key support) as he might have concerns/intuitions that we can't know about. I do not have Pypi

Re: [VOTE] Airflow 1.9.0rc4

@bolke/@fokko, thoughts? As an aside, we had an issue seeing task logs, but were able to fix it by modifying our logging config (and perhaps patching some stuff that we merged into 1.9.0, don't recall). William Pursell/Joy Gao might be able to comment more. On Thu, Nov 30, 2017 at 9:08 AM, Ash

Re: [VOTE] Airflow 1.9.0rc4

@Ash, do you think this issue is blocking for 1.9.0? On Thu, Nov 30, 2017 at 8:22 AM, Ash Berlin-Taylor < ash_airflowl...@firemirror.com> wrote: > Ah, I think I've got it. > > The problem is that airflow.task.raw doesn't have an explicit config > anymore(?), so when we do: > > log =

Re: [VOTE] Airflow 1.9.0rc4

Yes, that is in my logging config. I think I agree about airflow.task.raw -ash > On 30 Nov 2017, at 18:06, Joy Gao wrote: > > Hi Ash, > > Can you verify that your logging config has root logger set to 'console'? > Something like: > >'': { >'handlers':

Re: Enabling flake8

This is great! I've been following the guidelines here which indicate PEP8 linting, is this consistent with the new flake8 testing we will be doing? - Andy On Thu, Nov 30, 2017 at 10:58 AM, Bolke de

Re: RBAC Update

A bit of related history here: https://github.com/dpgaspar/Flask-AppBuilder/issues/399 On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Given I have merge rights on FAB I could probably do another round of > review and get your PRs through. I would really

Re: [VOTE] Airflow 1.9.0rc4

If anyone else is seeing this (and I think I'm on stock configuration now) yes I do, as I'm not seeing _any_ logs form task operators appearing. There's another problem after applying a local fix for `handler.set_context()`, to do with try_numbers: The `airflow run tests test-logging

Re: [VOTE] Airflow 1.9.0rc4

Ah, I think I've got it. The problem is that airflow.task.raw doesn't have an explicit config anymore(?), so when we do: log = logging.getLogger('airflow.task.raw') and look at its handlers, it doesn't have any. But log.parent.handlers _does_ have them. A few options to fix this: - I can

Re: Enabling flake8

It is enabled now for diffs. Strict, maximum line length 90. Looking forward to cleaner code ;-). Bolke. > On 28 Nov 2017, at 03:12, Maxime Beauchemin > wrote: > > A colleague recently did something similar for Superset. I'm really glad we > did. We started by

Re: [VOTE] Airflow 1.9.0rc4

We're running more tests (day of meetings meant we couldn't do as many as I'd have liked yesterday) but looking good so far. +0.5 (non binding) -ash > On 29 Nov 2017, at 18:31, Chris Riccomini wrote: > > Anyone else? Last day before I close the vote off. > > On Mon,

Re: Introducing a "LAUNCHED" state into airflow

See reset_state_for_orphaned_tasks in jobs.py On Thu, Nov 30, 2017 at 11:17 AM, Alex Guziel wrote: > Right now the scheduler re-launches all QUEUED tasks on restart (there are > safeguards for duplicates). > > On Thu, Nov 30, 2017 at 11:13 AM, Grant Nicholas

Re: [VOTE] Airflow 1.9.0rc4

Ok, I think I can confirm the issue. I don’t see any output either with a clean install. I’ll investigate a bit further. *sigh* Bolke > On 30 Nov 2017, at 19:13, Ash Berlin-Taylor > wrote: > > Yes, that is in my logging config. > > I think I agree about

Re: [VOTE] Airflow 1.9.0rc4

Ill have a patch in an hour or so > On 30 Nov 2017, at 19:28, Chris Riccomini wrote: > > K, I'm going to let this vote hang here until we get confirmation as to > what's going on. > > @bolke/@ash, thanks for looking into this! > > On Thu, Nov 30, 2017 at 10:23 AM,

Re: Introducing a "LAUNCHED" state into airflow

It can potentially add value, at least to the user. The user knows they may be undersubscribed on workers, whether those are k8s or something else. You'll want to think about how execution_timeout relates to this, right? In a different system I work on, we have: READY, QUEUED, SUBMITTED, and

Re: Introducing a "LAUNCHED" state into airflow

Right now the scheduler re-launches all QUEUED tasks on restart (there are safeguards for duplicates). On Thu, Nov 30, 2017 at 11:13 AM, Grant Nicholas < grantnicholas2...@u.northwestern.edu> wrote: > @Alex > I agree setting the RUNNING state immediately when `airflow run` starts up > would be

Re: [VOTE] Airflow 1.9.0rc4

Patch here: https://github.com/apache/incubator-airflow/pull/2831 Bolke > On 30 Nov 2017, at 20:53, Bolke de Bruin wrote: > > Ill have a patch in an hour or so > > >> On 30 Nov 2017, at 19:28, Chris Riccomini wrote: >> >> K, I'm going to let this

Re: [VOTE] Airflow 1.9.0rc4

K, I'm going to let this vote hang here until we get confirmation as to what's going on. @bolke/@ash, thanks for looking into this! On Thu, Nov 30, 2017 at 10:23 AM, Bolke de Bruin wrote: > Ok, I think I can confirm the issue. I don’t see any output either with a > clean

Re: Introducing a "LAUNCHED" state into airflow

@Alex That could potentially work since if you have the same task launched twice then the second time would die due to the "already running dependency". Still less ideal than not launching that task at all since it still allows for race conditions. @grant thoughts on this? On Wed, Nov 29, 2017

Re: Introducing a "LAUNCHED" state into airflow

I think the more sensible thing here is to just to set the state to RUNNING immediately in the airflow run process. I don't think the distinction between launched and running adds much value. On Thu, Nov 30, 2017 at 10:36 AM, Daniel Imberman wrote: > @Alex > > That

Re: Data lineage and data portal

Hi all, Nice overwhelming response! :) I'd like to host a meeting on zoom to discuss. I'm on the free plan there, so we are limited to 40 minutes! :) (msg me priv if you have better hosting options). Here's a way to let me know your availability without polluting the mailing list:

Re: [VOTE] Airflow 1.9.0rc4

And here's a patch for that one: https://github.com/apache/incubator-airflow/pull/2832 Any feedback I'll address in the morning (which is only 8 hours away.) Anyone is welcome to change this branch or if you have an idea for a better fix

Re: [VOTE] Airflow 1.9.0rc4

Thanks for picking that up so speedily Bolke! I've opened a jira ticket for the other problem I found: https://issues.apache.org/jira/browse/AIRFLOW-1873 and I'll come up with a quick patch for that now. (Quick in the sense that a fix I've

Re: Introducing a "LAUNCHED" state into airflow

Taking a tangent here: I like the idea of logging every state change to another table. Mutating task_instance from many places results in things that are hard to debug in some cases. As we need similar history-tracking of mutations on task_instances around retries, we may want keep track of

Re: [VOTE] Airflow 1.9.0rc4

Hmmm is anyone else seeing logs from inside operators? Right now we're not getting any logs out of operators. I'm in the process of debugging to work out if it's our logging config or something in Airflow, but right now I don't seem to get any output from the operator itself, neither print(),

Re: Kubernetes Executor

Hi Daniel, I would love get involved with this. Let me know how I can contribute. Best Best regards Niranda Perera Research Assistant Dept of CSE, University of Moratuwa niranda...@cse.mrt.ac.lk +94 71 554 8430 https://lk.linkedin.com/in/niranda On Wed, Nov 29, 2017 at 11:17 PM, Daniel

Re: [VOTE] Airflow 1.9.0rc4

airflow.task.raw points to airflow.task by default, which does have a handler and that is the one that is travelled. And yes we know the code is incorrect in this area, but the default does work correctly. Bolke. > On 30 Nov 2017, at 16:27, Ash Berlin-Taylor