Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

@Jarek & Fokko Firstly, thank you for the thorough and thoughtful replies -- I appreciate it. I recognize that the execution_date-based idempotence pattern is core to airflow. I understand that it motivated a lot of the design decisions, and that it is a pattern that is endorsed and explicitly

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

To banish anything stateful seems arbitrary and unnecessary. Airflow is more than its canonical task structure: hook / operator framework and ecosystem, scheduling, retry, alerting, distributed computing, etc etc etc etc. As long as support for the canonical task is preserved, what's the harm in

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

gt; Really good document Jacob. > > > > > > Below are my thoughts on different topics discussed in the docs and the > > > mailing list: > > > > > > > > > *Prefix on Xcom* > > > I don't think that is a good idea to mix this into Xcom. We

Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

I also am a big fan of adding better support for stateful tasks, though I know this is a thorny subject in airflow community. There are many data warehousing tasks where state makes a lot of sense. While idempotence is a nice design pattern it's not the solution for every problem. XCom may not

Re: Airflow DAG explodes with RecursionError when triggered

Did you check the max_active_runs param in your dag? On Mon, Dec 23, 2019, 11:47 PM Shubham Gupta wrote: > Hello maintainers > > A programmatically generated DAG of mine is exploding (the bomb > text-picture) when triggered with following stack-trace > > File >

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

Re For example S3 -> GCS should be in "google" provider, but GCS-> S3 should > be in "amazon > So if there were a BigQueryToS3 or SnowflakeToS3 operator, would you put this in AWS? I feel like storage should be a secondary consideration concerning object naming. Using snowflake as an example,

Re: Can an xcom push and pull to the same task instance on the same execution date?

In current releases xcom is deleted at task start. Recently a PR was merged to not delete until XCOM is again pushed I am not sure if this has made it into 1.10.6rc After this change, I think what you are trying to do would work. But it's

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

One case popped up for us recently, where it made sense to make a MsSql *From*S3Operator . I think using "source" makes sense in general, but in this case calling this a S3ToMsSqlOperator and putting it under AWS seems silly, even though you could say s3 is "source" here. I think in most of

Re: Setting to add choice of schedule at end or schedule at start of interval

le. Allowing for scheduling dags > at the left edge of an interval and allowing it to behave more like > cron, where it runs at the time specified, not schedule + interval, > would make things much less complicated for users like myself that > can't always wait until the right edge of the int

Re: Setting to add choice of schedule at end or schedule at start of interval

> a UI", so we should try to be pragmatic and fit in power-user features > where we can do so without compromising the project's overall goals. > > On Wed, Aug 28, 2019 at 2:24 PM Daniel Standish > wrote: > > > I am just thinking there is the potential for a more com

Re: DAG "Schedule Filter Callback"?

; simplistic and based around letting a sufficiently motivated user implement > the filtering logic themselves: > https://docs.prefect.io/guide/core_concepts/schedules.html#design > > Damian > > -Original Message- > From: Daniel Standish [mailto:dpstand...@gmail.com] > Sent

Re: DAG "Schedule Filter Callback"?

es the > prev_execution_date and next_execution_date to be the same as the execution > date as per this line of code: > > https://github.com/apache/airflow/blob/7a59358ffde269701af2121246ac54f1a5cbe785/airflow/models/taskinstance.py#L1129 > . > > Making "prev" and

Re: DAG "Schedule Filter Callback"?

Have you considered using TriggerDagOperator? One way to deal with this kind of thing is to have two dags: - "working dag" - This dag does the work. Its behavior is governed by execution_date / dag_run.conf. - "trigger dag" - This dag just triggers the "working" dag, with appropriate

Re: Airflow Dynamic tasks

be due to not having enough worker > slots. What > type of executor are you using, how is it configured? > > Max > > On Tue, Aug 20, 2019 at 7:50 AM Bacal, Eugene > > wrote: > > > Absolutely possible, Daniel, >

Re: Setting to add choice of schedule at end or schedule at start of interval

scheduling > logic inside the dag, when scheduling logic should be in the scheduler. > > -James > > > > On Aug 23, 2019, at 3:14 PM, Daniel Standish > wrote: > > > > Re > > > >> What are people's feelings on changing the default execution t

Re: Setting to add choice of schedule at end or schedule at start of interval

Re > What are people's feelings on changing the default execution to schedule > interval start and > I'm in favor of doing that, but then exposing new variables of > "interval_start" and "interval_end", etc. so that people write > clearer-looking at-a-glance DAGs While I am def on board with

Re: Airflow Dynamic tasks

It's not just webserver and scheduler that will parse your dag file. During the execution of a dag run, dag file will be re-parsed at the start of every task instance. If you have 1000 tasks running in short period of time, that's 1000 queries. It's possible these queries are piling up in a

Re: Airflow Dynamic Dags

If your dag generation process makes external calls over network, one avenue of optimization is to cache your config (i.e. the thing that drives your dag generation process) in local file (perhaps sqlite) and update that file only periodically. Or, just convert to fixed config object (e.g. a

Re: [DISCUSS] Tweaks to the Airflow logo

> > Maybe there is solid reasoning behind this change; if so, that reasoning > would be welcome. Oh this info is in the referenced pdf 臘‍♂️ On Tue, Aug 13, 2019 at 8:51 PM Daniel Standish wrote: > It would probably be prudent to review how it looks in different > situations, e.

Re: [DISCUSS] Tweaks to the Airflow logo

It would probably be prudent to review how it looks in different situations, e.g. as a favicon on chrome, and as a small icon in airflow home page. Perhaps Chris could provide? Additionally I am sympathetic to James's concerns (which somehow ended up in a different thread). I like the solid

Re: [VOTE] Changes in import paths

; > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > > >>>>>>> > > >>>>>>> in the Wiki - which I should have mentioned before. I have not > much > > >>>> time > > >>>>>>> today - but tomorrow I

Re: [VOTE] Changes in import paths

I have tried to add some clarification to Jarek's summary, but I am a little fuzzy on exact proposal for case 3 so hopefully Jarek can further clarify. According to my reading, in this motion cases 4,5,6 are all either proposal *rejections* or otherwise have no effect, so attention can be focused

Re: [DISCUSS] AIRFLOW-4192 - remove duplicate/obsolete/derived task context variables

I would like to add to this discussion a change to `TaskInstance.previous_ti`. It is a useful property in template context, but it has an identity crisis. Currently this property may return a different value depending on whether your dag is `catchup` and whether your dag has a schedule interval.

Re: Bay Area Airflow meetup

Per Feng, the presentations will be recorded and links published to dev list / meetup page, but will not be live streamed. On Tue, Jun 4, 2019 at 4:07 PM Teresa Martyny < teresa.mart...@omadahealth.com> wrote: > Recording/streaming the meetup talks every time would be amazing for folks > like me

Re: Tasks that run just once

> > However, I also want to create the tables themselves via Airflow and run a > one time backfill for each table. Backfilling hourly is proving to be a > very inefficient process. > So, one thing, I have implemented alternative "initial load" behavior inside the execute method of an operator by

Re: scheduler logging

> Can you confirm if that log message is actually affecting behaviour, or > does it just look bad? > > -ash > > On 23 April 2019 19:38:57 BST, Daniel Standish > wrote: > >I noticed two issues in 1.10.2, one of which has been resolved, but I > >am

Re: scheduler logging

I noticed two issues in 1.10.2, one of which has been resolved, but I am not sure if they were present in 1.10.0, or if they are the messages that are bothering you. 1. "Harvesting DAG parsing results" was printed every 2 seconds or something. This was resolved in commit [AIRFLOW-3911] Change

Re: Hangout: Learn how to hack on Airflow

Me too please. On Wed, Mar 27, 2019 at 7:31 AM Martin Gauthier wrote: > Hi. I'm in please add me !! Thank you > > On Wed, Mar 27, 2019 at 10:18 AM Stachurski, Stephan < > stephan.stachur...@nytimes.com> wrote: > > > Please add me! > > > > On Wed, Mar 27, 2019 at 8:24 AM Thao Ha Thi Phuong > >

<    1   2   3