Aligning CLI / UI for 'clear' and 'mark success'

Hi all, I'm working on aligning features ('clear', 'mark success') between CLI and UI and would like some feedback on this issue: https://issues.apache.org/jira/browse/AIRFLOW-184 Thanks! Joy

Re: DagRun date "off by one"

+1 on this. We are using Airflow as a cron replacement, and we have biweekly jobs and monthly jobs as well. It would be really useful to be able to configure it such that dags run on the start_date and the timestamps corresponds to it. On Tue, Jul 19, 2016 at 10:49 AM, Tyrone Hinderson wrote: >

Re: Role Based Access Control for Airflow UI

;> I think it would also be good to think about this > > > proposal/implementation > > > >>>> and how it applied in the API-driven world (e.g. when webserver > hits > > > APIs > > > >>>> like /clear on behalf of users instead of running commands against > > the > > > >>>> database directly). > > > >>>> > > > >>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin < > bdbr...@gmail.com > > > > > > >>>> wrote: > > > >>>> > > > >>>>> Will respond but im traveling at the moment. Give me a few days. > > > >>>>> > > > >>>>> Sent from my iPhone > > > >>>>> > > > >>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini < > criccom...@apache.org> > > > >>>> wrote: > > > >>>>>> > > > >>>>>> Hey all, > > > >>>>>> > > > >>>>>> Checking in on this. We spent a good chunk of time thinking > about > > > >>> this, > > > >>>>> and > > > >>>>>> want to move forward with it, but want to make sure we're all on > > the > > > >>>> same > > > >>>>>> page. > > > >>>>>> > > > >>>>>> Max? Bolke? Dan? Jeremiah? > > > >>>>>> > > > >>>>>> Cheers, > > > >>>>>> Chris > > > >>>>>> > > > >>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar < > > > >>>>>> kalpeshdharwad...@gmail.com> wrote: > > > >>>>>> > > > >>>>>>> Hello everyone, > > > >>>>>>> > > > >>>>>>> As you all know, currently Airflow doesn’t have a built-in Role > > > >>> Based > > > >>>>>>> Access Control(RBAC) capability. It does provide very limited > > > >>>>>>> authorization capability by providing admin, data_profiler, and > > > user > > > >>>>> roles. > > > >>>>>>> However, associating these roles to authenticated identities is > > not > > > >>> a > > > >>>>>>> simple effort. > > > >>>>>>> > > > >>>>>>> To address this issue, I have created a design proposal for > > > building > > > >>>>> RBAC > > > >>>>>>> into Airflow and simplifying user access management via the > > Airflow > > > >>>> UI. > > > >>>>>>> > > > >>>>>>> The design proposal is located at https://cwiki.apache.org/ > > > >>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal > > > >>>>>>> > > > >>>>>>> Any comments/questions/feedback are much appreciated. > > > >>>>>>> > > > >>>>>>> Thanks > > > >>>>>>> Kalpesh > > > >>>>>>> > > > >>>>> > > > >>>> > > > >>> > > > > > > > > > > > > > -- Joy Gao Software Engineer 350 Convention Way, Suite 200 Redwood City, CA 94063 Mobile: 669-224-9305 Payments partner to the platform economy

Re: Role Based Access Control for Airflow UI

my fork, I went far enough to > >>>> hit > >>>> > > some > >>>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow > >>>> > > > Bootstrap > >>>> > > > > > >> template too so t

Re: DAG with ExternalTaskSensor won't run after deletion

Hi Ali, I was able to replicate your scenario above. The executor maintains a cache of running task instances. When a task instance finishes, it is removed from the `running` cache of the executor. When you hard-delete the DAG Run and the task instances via the UI, the corresponding task instance

Re: Terminate task process through UI

;> > >>> wrote: > >>> > >>>> Hello, > >>>> > >>>> Is there a way to trigger the on_kill() method of an operator through > the > >>>> UI? > >>>> I tried changing the state of a running task and deleting it, but > neither > >>>> approach triggered the on_kill() method and so the process was still > >>>> running. > >>>> > >>>> Similar question was asked on SO: > >>>> https://stackoverflow.com/questions/43626844/kill-the- > >>>> airflow-task-running-on-a-remote-location-through-airflow-ui > >>>> > >>>> Richard > >>>> > >>> > > > > -- Joy Gao Software Engineer 350 Convention Way, Suite 200 Redwood City, CA 94063 Mobile: 669-224-9305 Payments partner to the platform economy

Re: Meetup Interest?

Hi Dan, I'd be happy to give an update on progress of the new RBAC UI we've been working on here at WePay. Cheers, Joy On Fri, Oct 13, 2017 at 12:10 PM, Dan Davydov < dan.davy...@airbnb.com.invalid> wrote: > Is there interest in doing an Airflow meet-up? Airbnb can host one in San > Francisco.

Re: programatically configure aws_default, emr_default connections?

Just want to add that a PR ( https://github.com/apache/incubator-airflow/pull/2525) is underway to allow adding connections via CLI (bypassing the uri schema issue). On Wed, Oct 18, 2017 at 11:45 AM, Frank Maritato wrote: > Thanks, Andrew! This works nicely. > > On 10/16/17, 3:01 PM, "Andrew Che

Re: Scheduler Keeps Crashing

I've had something similar happening to me in the past (`airflow upgradedb` stuck forever waiting for a lock on a model). Because some migration scripts require querying the db, it got stuck waiting for the lock on a table. For me it had to do with calling a model's class method in a dag file, whi

Re: Multiple similar DAGs

Hi Michael, You could leverage the Variable feature in Airflow: start by setting a different environment variable in each env, and then in your DAG file, instead of hard-coding the schedule, call a method that looks up which environment it is (using Variable.get) and returns the desired one. Hope

Re: new logging

Hi Kevin, Do you have a root logger defined in the logging config? And if so, make sure it's logging to 'console' so that base_task_runner can pick up the logs via stdout. See this part of the default config: https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_

RBAC Update

Hi guys. I've been working on moving airflow from Flask-Admin to Flask-AppBuilder for RBAC , check it out at https://github.com/wepay/airflow-webserver. It's still a work-in-progress, but most features you see in the webse

Re: RBAC Update

> I know the weekend is coming up, and for those of us in the US, > next > > > week > > > > > is a bit of a slow holiday week. Would love to get some feedback > from > > > > > everyone on this. The goal would ideally to be to converge

Re: RBAC Update

ly. Yes, this means there will be a short period of having to maintain two UIs, but once we freeze development on www, it shouldn't be a concern for long.* I'd love to hear everyone's thoughts on this! I'm excited about bringing RBAC to airflow and I hope it's something

Re: RBAC Update

rname: viewer password: password On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao wrote: > Hi folks, > > Thanks for all the feedback regarding to the new Airflow Webserver UI > <https://github.com/wepay/airflow-webserver/>! I've been actively > addressing all the bugs that were

Re: [VOTE] Airflow 1.9.0rc4

mijn iPad > > > Op 30 nov. 2017 om 18:17 heeft Chris Riccomini > het volgende geschreven: > > > > @bolke/@fokko, thoughts? > > > > As an aside, we had an issue seeing task logs, but were able to fix it by > > modifying our logging config (and perhaps patc

Re: RBAC Update

at. > > > > I've threatened to fork the project before, that's always an option. I've > > noticed his involvement is sporadic and comes in bursts. > > > > In the meantime, you can have the dependency in Airflow Webserver > pointing > > straight

Re: Environment variables - Oracle LD_LIBRARY_PATH

Hi Rajesh, `airflow test` executes the task synchronously in the same python process (without touching the scheduler), which is probably why it went down the happy path. Are you using celery executor and running the tasks in a distributed manner? It* sounds like* the task execution env doesn't kn

Re: Environment variables - Oracle LD_LIBRARY_PATH

set the > environment variable, but it does not work either. > > > > On Dec 8, 2017, at 2:50 PM, Joy Gao wrote: > > > > Hi Rajesh, > > > > `airflow test` executes the task synchronously in the same python process > > (without touching the schedul

Re: [VOTE] Airflow 1.9.0rc6

+1, binding :) On Thu, Dec 14, 2017 at 3:25 AM, Bolke de Bruin wrote: > +1, binding > > Verstuurd vanaf mijn iPad > > > Op 13 dec. 2017 om 20:37 heeft Trevor Edwards > > > het volgende geschreven: > > > > +0.5 > > > > So far looks good to me. There is a slight loss in backwards > compatibility

Re: [VOTE] Airflow 1.9.0rc8

+1, binding Thank you Chris! On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini wrote: > Hey all, > > (Last time, I hope)^2 > > I have cut Airflow 1.9.0 RC8. This email is calling a vote on the release, > which will last for 72 hours. Consider this my (binding) +1. > > Airflow 1.9.0 RC8 is availa

Re: [VOTE] Airflow 1.9.0rc8

Bruin > > > > > +1, binding > > > > > > Checked sigs, version, source is there (did not check build), bin is > > there. > > > > > > Bolke > > > > > > Verstuurd vanaf mijn iPad > > > > > > > Op 15 dec.

Q1 Airflow Bay Area Meetup

Hi folks, I'm Joy from WePay. At the last Airflow meetup in December there was a demand for hosting a future meetup to cover topics on Airflow integrations with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best practices on running Airflow in the cloud, managed Airflow, etc.) If ther

Re: Q1 Airflow Bay Area Meetup

you to *Event Organizer* on the Bay Area Apache > Airflow > > > > > Meetup as per instructions/guidelines on > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups > > > > > > > > > > Go ahead and start setting up the m

Re: Airflow 1.10

The new FAB UI does not modify the existing API (i.e. www2/api/ will be a copy of www/api/), and the endpoints are registered as blueprints to the flask app the same way as before, so it is fully backward compatible. Although FAB offers REST APIs on the models out-of-the-box, it currently is still

Re: Fix on_kill command for operators

@Milan, I agree this bug needs to be fixed, and the the hack I provided previously isn't complete as it doesn't work for any operator with on_kill method that requires transient task attributes (i.e. self.sp)... Unfortunately I don't think the new UI will help rid of the issue (as the bug is on the

Re: RBAC Update

Hi folks, I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out for the new UI. I've included instructions on how to test it out in the PR description. Looking forward to your feedbacks. Cheers, Joy On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao wrote: > Thanks for

Re: Help with podling status report?

Thanks Sid! On Thu, Feb 8, 2018 at 11:21 AM, Sid Anand wrote: > Updated. > > https://wiki.apache.org/incubator/February2018 > > -s > > On Thu, Feb 8, 2018 at 11:13 AM, Feng Lu > wrote: > > > Thank you Sid! > > > > > > On Thu, Feb 8, 2018 at 10:41 AM Sid Anand wrote: > > > > > I can take care o

Re: Q1 Airflow Bay Area Meetup

tion behavior >> >- how I would have used Airflow if I'd known then what I know now >> > >> > >> > On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire >> > wrote: >> > >> > > +1 on being able to watch a recording, based i

April Airflow Meetup (Details Finalized)

Hi everyone, WePay will be hosting the next Airflow meetup on Wednesday, April 11th. I've updated the event itinerary on the meetup event page . We've curated a set of talks that are infra- and SRE-oriented for t

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Congrats and welcome! :D On Fri, Feb 23, 2018 at 8:27 AM, Sid Anand wrote: > Folks! > Please join the Airflow PPMC in welcoming Ash Berlin-Taylor to its ranks as > both committer and PPMC member! Congrats Ash! > > Announcement : > https://cwiki.apache.org/confluence/display/AIRFLOW/ > Announcem

Re: How to add hooks for strong deployment consistency?

Hi David, Do you mind providing a concrete example of the scenario in which scheduler/workers see different states (I'm not 100% sure if I understood the issue at hand). And by same dag generation, are you referring to the dag version? (DAG version is currently not supported at all, but I can see

Re: How to add hooks for strong deployment consistency?

there isn't a way to do > this other than turn off the DAG, and login to all hosts to verify fully > deployed. > > We would like to remove this confusion and make generations/versions (same > thing really) exposed to users and make sure for a single dag_run only one > version is used

Re: How to add hooks for strong deployment consistency?

t;> > On Tue, Feb 27, 2018 at 6:48 PM, David Capwell >> wrote: >> > >> > > Thanks for your feedback! >> > > >> > > Option 1 is a non-starter for us. The reason is we have DAGs that take >> 9+ >> > > hours to run. >> &g

Re: [VOTE] Migrate to Github as primary repo (a.k.a. Gitbox)

+1 (binding) On Mon, Mar 12, 2018 at 10:23 AM, Chris Riccomini wrote: > +1 > > On Sat, Mar 10, 2018 at 11:18 PM, Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > +1 (binding) > > > > On Sat, Mar 10, 2018 at 2:03 PM, Beau Barker > > wrote: > > > > > +1 for Github. > > > > > > Also

Re: Custom DagBag collection based on dags_folder prefix

Hi guys, A related topic has been discussed recently via a separate email thread (see 'How to add hooks for strong deployment consistency? ') The idea brought up by Maxime is to modify

Re: RBAC Update

t the RBAC UI will be able to bring additional security to Airflow, and with FAB framework in place we can look into leveraging it for a unified set of APIs used by both UI and CLI. Joy On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao wrote: > Hi folks, > > I have a PR <https://github.com/a

Re: DAG Level permissions (was Re: RBAC Update)

houghts? >> >> Max >> >> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng wrote: >> >> > Great work Joy. This is awesome! I am interested in helping out the per >> dag >> > level access. Just created a ticket to check(AIRFLOW-2267). Let me >

Re: DAG Level permissions (was Re: RBAC Update)

as an extra method "can_dag_action(user, > dag_id, action)". The SecurityManger class is neat because people can > provide their own if they want and override methods. That means that you > can defer any of the security-related checks to another system if you want > to. Many co

Re: Give it up for Fokko!

🙌👍 On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil wrote: > Couldn't agree more. Thanks Fokko > > On 13/04/2018, 17:56, "Maxime Beauchemin" > wrote: > > Hey all, > > I wanted to point out the amazing work that Fokko is doing, > reviewing/merging PRs and doing fantastic committer & ma

Re: Apache Airflow welcome new committer/PMC member : Naik Kaxil (a.k.a. kaxil)

Welcome! :) On Tue, May 8, 2018 at 2:02 PM, Victor Noagbodji < vnoagbo...@amplify-nation.com> wrote: > Welcome Naik! > > > On May 8, 2018, at 4:47 PM, Driesprong, Fokko > wrote: > > > > Hi Airflow'ers, > > > > Please join the Apache Airflow PMC in welcoming its newest member and > > co-committer

Re: Python3 and sensors module

Not a full patch to replace snakebite with libhdfs, but should temporarily unblock ci/cd: https://github.com/apache/incubator-airflow/pull/3365 On Wed, May 16, 2018 at 10:00 AM, Bolke de Bruin wrote: > Snakebite is not py3 compatible. We should move to libhdfs. Happy to take > patches! > > B. >

Re: [VOTE] Airflow 1.10.0rc2

Tested this with new installment of Airflow in python 2.7/3.6, mainly focusing on: - simple DAGs running with local executor - log generation / rendering - fernet key generation on installment - rbac/non-rbac UI view rendering Voting: +1 (binding) On Mon, Jul 16, 2018 at 1:20 AM, Naik Kaxil w