Re: Deprecate legacy UI in favor of FAB RBAC

The branch has for 1.10 is different from master (v1-10-test and stable) so there is no issue from that perspective. If there are fixes for 1.10 after the merge that concern the old ui they can just be based against the right branch. B. Op di 18 dec. 2018 13:49 schreef airflowuser I don't think

Re: [RESULT] Graduate Apache Airflow as a TLP

iel, Shah, Stefan, Kevin, Marc, Sunil, > > >> > Adityan, Deng, Neelesh, Sai > > >> > > > >> > I'll use this result to start the corresponding VOTE on the IPMC. > I'm > > >> > at an offsite today, so I have limited email time.

Re: [VOTE] Graduate the Apache Airflow as a TLP

;>>> related to workflow automation and scheduling that can be >>>> used to author and manage data pipelines; and be it further >>>> >>>> RESOLVED, that the office of "Vice President, Apache Airflow" be >>>> and hereby is

Re: [DISCUSS] Apache Airflow graduation from the incubator

Nov 28, 2018 at 2:48 PM Tao Feng wrote: >>> >>> Sorry, just saw Kaxil's latest email. Kaxil, is there anything else I could >>> help with? >>> >>> Thanks, >>> -Tao >>> >>>> On Wed, Nov 28, 2018 at 2:40 PM Tao

Re: [VOTE] Graduate the Apache Airflow as a TLP

irection of the Board of Directors as the chair >> of the Apache Airflow Project, and to have primary responsibility >> for management of the projects within the scope of >> responsibility of the Apache Airflow Project; and be it further >> >> RESOLVED, that the pe

Re: [DISCUSS] Apache Airflow graduation from the incubator

Ping! Verstuurd vanaf mijn iPad > Op 27 nov. 2018 om 21:39 heeft Bolke de Bruin het > volgende geschreven: > > Hi Folks, > > Thanks all for your responses and particularly Stefan for his suggestion to > use the generic Apache way to handle security issues. This seems

Re: [DISCUSS] Apache Airflow graduation from the incubator

ning of this year, but the project did two >>> releases during that time, once TLP releasing becomes easier :) >>> >>> Regarding QU30 you may consider to use the ASF wide security mailing >>> list [3] and process [4]. >>> >>> Kind Regards

Re: [DISCUSS] Apache Airflow graduation from the incubator

ng of this year, but the project did two >> releases during that time, once TLP releasing becomes easier :) >> >> Regarding QU30 you may consider to use the ASF wide security mailing >> list [3] and process [4]. >> >> Kind Regards, >> Stefan >> >

Re: [DISCUSS] Apache Airflow graduation from the incubator

Ping! Sent from my iPhone > On 24 Nov 2018, at 12:57, Bolke de Bruin wrote: > > Hi All, > > With the Apache Airflow community healthy and growing, I think now would be a > good time to > discuss where we stand regarding to graduation from the Incubator, and what &

[DISCUSS] Apache Airflow graduation from the incubator

Hi All, With the Apache Airflow community healthy and growing, I think now would be a good time to discuss where we stand regarding to graduation from the Incubator, and what requirements remains. Apache Airflow entered incubation around 2 years ago, since then, the Airflow community learned

Re: Remove airflow from pypi

Agree! This is even a security issue. Sent from my iPhone > On 23 Nov 2018, at 15:29, Driesprong, Fokko wrote: > > Hi all, > > I think we should remove airflow (not > apache-airflow) from Pypi. I still get questions from people who > accidentally install Ai

Re: Airflow 1.10.1 is released

That was pretty smooth sailing! Well done! Sent from my iPhone > On 22 Nov 2018, at 04:13, Kevin Yang wrote: > > 🎉🎉 Thank you Ash for cutting the release! > >> On Wed, Nov 21, 2018 at 6:09 PM Sid Anand wrote: >> >> Excellent work Ash! Thanks for doing the needful!! >> >> -s >> >>> On Wed,

Re: [VOTE CANCELED] Airflow 1.10.1rc1

You need to wait 72h and until you have 3+ votes. Satisfy both conditions :-) On putting things on pypi I'm personally in favor. It's not an Apache channel thus not official. On the other hand Apache is Java focused and only understands SNAPSHOTS but that's essentially the same as this. B. S

Re: Airflow 1.10.1b1 release available - PLEASE TEST

Please make sure not to call it release at least 🙂 Sent from my iPhone > On 12 Nov 2018, at 22:51, Ash Berlin-Taylor wrote: > > Hi Hitesh, > > My understanding was that the only official place an Apache release can > happen from is https://www.apache.org/dist/incubator/airflow/ >

Re: 1.10.1 Release?

The fix is in master and should work across all DST changes. It will be included in 1.10.1. B. Sent from my iPhone > On 5 Nov 2018, at 19:54, Dave Fisher wrote: > > > >> On 2018/10/28 00:09:05, Bolke de Bruin wrote: >> I wonder how to treat this: >> &

Re: 1.10.1 Release?

They are :-) I might have to adjust some tests, let's see. Will do that tomorrow if required. B. > On 30 Oct 2018, at 21:53, Ash Berlin-Taylor wrote: > > Fair :) Timezones are _hard_ > > Giving it a look now. > > -ash > >> On 30 Oct 2018, at 20:50, Bolke

Re: 1.10.1 Release?

> Ash > > On 30 October 2018 19:40:27 GMT, Bolke de Bruin wrote: > I think we should use the UTC date for cron instead of the naive local date > time. I will check of croniter implements this so we can rely on that. > > B. > > On 28 Oct 2018, at 02:09, Bolke de Bruin w

Re: 1.10.1 Release?

Patch available at: https://github.com/apache/incubator-airflow/pull/4117 please test. B. > On 30 Oct 2018, at 21:14, Bolke de Bruin wrote: > > We should just pass it the UTC date (we should never use local time except at > the user interface). I’m testing a patch right

Re: 1.10.1 Release?

ll give 2:55 CEST followed by 2:00 CET) and then we convert it to > UTC for return - but right now we are giving it a TZ-unaware local time. > > I think. > > Ash > > On 30 October 2018 19:40:27 GMT, Bolke de Bruin wrote: > I think we should use the UTC date for cron ins

Re: 1.10.1 Release?

I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that. B. > On 28 Oct 2018, at 02:09, Bolke de Bruin wrote: > > I wonder how to treat this: > > This is what I think happens (need to ver

Re: 1.10.1 Release?

gt; unblock the scheduler. > > Ash > > On 30 October 2018 18:43:30 GMT, David Klosowski wrote: >> Hi Airflow Devs: >> >> Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1? >> >> Thanks. >> >> Regards, >> David >>

Re: 1.10.1 Release?

ocalize(datetime(2017, 3, 26)) >>>> val = croniter('0 0 * * *', local_date).get_next(datetime) > > I think the problem is that we are _not_ passing a TZ aware dag in and we > should be. > >> On 30 Oct 2018, at 17:35, Bolke de Bruin wrote: >> >

Re: 1.10.1 Release?

rk !: >>>> local_date = tz.localize(datetime(2017, 3, 26)) >>>> val = croniter('0 0 * * *', local_date).get_next(datetime) > > I think the problem is that we are _not_ passing a TZ aware dag in and we > should be. > >> On 30 Oct 2018, at 17:3

Re: 1.10.1 Release?

>> to parse the dag... and it never returns. >> >> I've only just managed to reproduce it, so haven't dug in to why yet. A >> quick hacky debug print shows something is stuck in an infinite loop. >> >> -ash >> >>> On 29 Oct 201

Re: explicit_defaults_for_timestamp for mysql

the changes have been working for us so far. > > On Tue, Oct 23, 2018 at 10:09 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > We only need it at table creation time or alter table time during which an > alembic script would fail if MySQL restarts I assume? > > I&

Re: 1.10.1 Release?

it. > > Bolke: are there any outstanding tasks/issues that you know of that might > slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing > issues that were asked of us? I thought I read something about license > declarations in markdown files?) > >

Re: 1.10.1 Release?

> > > ‐‐‐ Original Message ‐‐‐ > On Friday, October 26, 2018 6:12 PM, Bolke de Bruin wrote: > >> Hey Ash, >> >> I was wondering if you are picking up the 1.10.1 release? Master is speeding >> ahead and you were tracking fixes for 1.10.1 right? >> >> B. > >

1.10.1 Release?

Hey Ash, I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right? B.

Re: explicit_defaults_for_timestamp for mysql

able=False) > -op.alter_column('xcom', 'execution_date', > existing_type=mysql.TIMESTAMP(fsp=6), nullable=False) > -op.alter_column('xcom', 'timestamp', > existing_type=mysql.TIMESTAMP(fsp=6), nullable=False) > - > +

Re: explicit_defaults_for_timestamp for mysql

O and reading https://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp indicates that it can be set on the session level as well. So

Re: explicit_defaults_for_timestamp for mysql

Please test this indeed. As mentioned I have tested this extensively and I could not make it work without. A possible workaround for MySQL could be is that we revert to “plain” datetimes and assume MySQL always runs in UTC. In addition we could have the converter take care of any mismanagement

Re: Ingest daily data, but delivery is always delayed by two days

S3 Bucket notification that triggers a dag? Verstuurd vanaf mijn iPad > Op 12 okt. 2018 om 12:42 heeft Ash Berlin-Taylor het > volgende geschreven: > > A lot of our dags are ingesting data (usually daily or weekly) from > suppliers, and they are universally late. > > In the case I'm setting

Re: execution_date - can we stop the confusion?

I dont think this makes sense and I dont that think anyone had a real issue with this. Execution date has been clearly documented and is part of the core principles of airflow. Renaming will create more confusion. Please note that I do think that as an anonymous user you cannot speak for any

Re: Airflow: Apache Graduation

y agenda, > where the Board approves it. Project is officially TLP once that is > done. > 5) Champion cleans up incubator resources, creates new TLP resources. > Example: https://jira.apache.org/jira/browse/INFRA-10639 > > -Jakob > > > On 21 September 2018 at 13:49, Bolke de

Re: Airflow: Apache Graduation

There is a maturity check that we can do. Don't have the link ready now. But can look it up later. To me two things need to be more mature 1. Handling of security issues 2. Licensing. 2a having ppl aware that with new dependencies updates to dependencies the respective licenses need to be update

Re: Sep Airflow Bay Area Meetup @ Google

gt; > On Fri, Sep 14, 2018 at 11:42 PM Feng Lu wrote: > >> Not going to happen for this time, we don't receive enough interest from >> the community. >> >> >> On Wed, Sep 12, 2018 at 7:57 AM Bolke de Bruin wrote: >> >>> btw how are we d

Re: Database referral integrity

Adding these kind of checks which work for integrity well make database access pretty slow. In addition it isnt there because in the past there was no strong connection between for example tasks and dagruns, it was more or less just coincidental. There also so some bisecting tools that probably

Live Airflow Meetup @ Amsterdam

https://www.youtube.com/watch?v=MM8tfTrcnfk Enjoy! Bolke

Re: Sep Airflow Bay Area Meetup @ Google

btw how are we doing on the “one day” hackathon? > On 12 Sep 2018, at 16:49, Bolke de Bruin wrote: > > Hi feng, > > I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam > (that I did together with Fokko). > > Cheers > Bolke > >&

Re: Sep Airflow Bay Area Meetup @ Google

Hi feng, I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam (that I did together with Fokko). Cheers Bolke > On 4 Sep 2018, at 22:13, Feng Lu wrote: > > We are 3 weeks away from the meetup and still have a few lightening talks > open, please take the chance and share

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

gt; On Mon, Sep 10, 2018 at 2:10 PM, Bolke de Bruin wrote: > >> I cannot connect this to timezone issues yet. The trigger dagrun operator >> seems to work fine (checked the examples). >> >> B. >> >>> On 10 Sep 2018, at 08:19, Deng Xiaodong wrote

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

I cannot connect this to timezone issues yet. The trigger dagrun operator seems to work fine (checked the examples). B. > On 10 Sep 2018, at 08:19, Deng Xiaodong wrote: > > Hi Goutam, > > Seems it’s due to the timezone setting? > > If that’s the case, you may try either adjusting your timezo

Re: Call for fixes for Airflow 1.10.1

che/incubator-airflow/pull/3693 > > > > > > This is a bug in terms of metadata, which also affects the UI > > “Browse->Jobs”. > > > > > > Thanks. > > > > > > Regards, > > > > XD > > > > On Wed, Sep 5, 2018 at 23

Re: [VOTE][RESULT] Replace with Gitter with Slack?

e > >- Sid Anand > > 0: 2 votes > >- Bolke de Bruin >- Kaxil Naik > > -0.5: 1 vote > >- Arthur Wiedmer > > > Vote result is a net positive of +0.5. > > I counted all of the PMC/committers' votes as binding. > > -s > >

Re: [VOTE][RESULT] Replace with Gitter with Slack?

gt; >>> +1: 12 votes >>> >>> - Sid Anand >>> - Steve Carpenter >>> - Beau Barker >>> - Marc Bollinger >>> - Pedro Machado >>> - Scott Halgrim >>> - Eamon Keane >>> - Adam Boscarino >>> - Danie

Re: [VOTE][RESULT] Replace with Gitter with Slack?

t;- Pedro Machado >- Scott Halgrim >- Eamon Keane >- Adam Boscarino >- Daniel Cohen >- Chandu Kavar >- William Horton >- Ben Gregory > > 0: 2 votes > >- Bolke de Bruin >- Kaxil Naik > > > -0.5: 1 vote > >-

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

Can you share the definition? And timezone settings? Op vr 7 sep. 2018 20:53 schreef Maxime Beauchemin < maximebeauche...@gmail.com>: > Is the issue timezone related? Personally I've only used Airflow in > UTC-aligned environments so I can't help much on this topic. Bolke as > contributed timezon

Re: Security issue being ignored?

Both are not security vulnerabilities: either it is in an upstream project or it is due to the way Airflow can be used. PR is welcome for the second JIRA. B. Verstuurd vanaf mijn iPad > Op 6 sep. 2018 om 11:07 heeft airflowuser > het volgende geschreven: > > Another example: > https://issues

Re: [VOTE] Replace with Gitter with Slack?

0 (binding) abstaining from the vote. I am not convinced we gain anything except disruption for an existing community. We cannot even force people to move away only make it “less” official. My question would be do we get more attendance of contributors and committers if we move to slack? Other

Re: Call for fixes for Airflow 1.10.1

You should push these to v1-10-test not to stable. Only once we start cutting RCs you should push to -stable. See the docs. This ensures a stable “stable”branch. Cheers Bolke. > On 3 Sep 2018, at 14:20, Ash Berlin-Taylor wrote: > > Hi everyone, > > I'm starting the process of gathering fixes

Re: Add git tag for 1.10

I’m not sure what you think is off? The change is only in master not in 1.10. If you are reading the docs for updating to master then you are using the wrong docs if you are upgrading to 1.10. B. Verstuurd vanaf mijn iPad > Op 3 sep. 2018 om 14:53 heeft Robin Edwards het volgende > geschreve

Re: Running unit tests against SLUGIFY_USES_TEXT_UNIDECODE and AIRFLOW_GPL_UNIDECODE (also is this broken?)

The dependency is so small, besides python-slugify executes tests with both that I wouldn’t bother. Probably it makes more sense to get rid of python-nvd3 and to be able to upgrade d3. Verstuurd vanaf mijn iPad > Op 28 aug. 2018 om 22:18 heeft Taylor Edmiston het > volgende geschreven: > >

Re: PR Review Dashboard?

I’m not in favor of moving to GitHub issues. While JIRA is not perfect it actually moves discussion to the mailing list. With GitHub issues the stuff just gets lost imho. I do like our changelogs a lot better now. B. Verstuurd vanaf mijn iPad > Op 27 aug. 2018 om 05:53 heeft Holden Karau het

Re: Python 3.6 Support for Airflow 1.10.0

Let’s not drop 2.7 too quickly but maybe mark it deprecated. I’m pretty sure Airbnb still runs on 2.7. Also RedHat does not deliver python 3 in its enterprise edition yet by default so it will put enterprise users in a bit of an awkward spot. B. Verstuurd vanaf mijn iPad > Op 28 aug. 2018 om

Re: Cloudera Hue in License

Yes we do. The kerberos ticket renewer i amended from hue. Sent from my iPhone > On 24 Aug 2018, at 17:46, Taylor Edmiston wrote: > > For anyone else who's curious - it looks like Hue is a web app for Hadoop. > > https://github.com/cloudera/hue > > https://en.wikipedia.org/wiki/Hue_(Hadoop)

Re: [RESULT][VOTE] Release Airflow 1.10.0

t;> wrote: >>>> >>>>> Thanks Bolke! >>>>> >>>>> I've just pushed the artifacts to Apache Dist: >>>>> >>>>> https://dist.apache.org/repos/dist/release/incubator/ >>>> airflow/1.10.0-incubating/

Fwd: [RESULT][VOTE] Release Airflow 1.10.0

iPhone Begin forwarded message: > From: Bolke de Bruin > Date: 20 August 2018 at 20:00:28 CEST > To: gene...@incubator.apache.org, dev@airflow.incubator.apache.org > Subject: [RESULT][VOTE] Release Airflow 1.10.0 > > The vote to release Airflow 1.10.0-incubating, having been op

[RESULT][VOTE] Release Airflow 1.10.0

message: > From: Bolke de Bruin > Date: 20 August 2018 at 19:56:23 CEST > To: gene...@incubator.apache.org > Subject: Re: [VOTE] Release Airflow 1.10.0 (new vote based on rc4) > > Appreciated Hitesh. Do you know how to add headers to .MD files? There seems > to be no technica

Fwd: [VOTE] Release Airflow 1.10.0 (new vote based on rc4)

Hi Folks, If you have contacts with IPMC members that can vote binding, can you please reach out to them and ask for help? The vote is stalled at the moment and without 3 +1's we can't release Airflow 1.10.0 Cheers Bolke -- Forwarded message ------ From: Bolke de Bruin

Re: apache-airflow v1.10.0 on PyPi?

/cv/taylor> | LinkedIn >> <https://www.linkedin.com/in/tedmiston/> | AngelList >> <https://angel.co/taylor> | Stack Overflow >> <https://stackoverflow.com/users/149428/taylor-edmiston> >> >> >> On Thu, Aug 9, 2018 at 5:43 PM, Krish Sigler wrote:

[RESULT][VOTE] Airflow 1.10.0rc4

Hello, Apache Airflow (incubating) 1.10.0 (based on RC4) has been accepted. 4 “+1” binding votes received: - Ash Berlin-Taylor (binding) - Naik Kaxil (binding) - Bolke de Bruin (binding) - Fokko Driesprong (binding) 1 “-1” non-binding: - Daniel Imberman (non-binding) My next step is to open

Re: [VOTE] Airflow 1.10.0rc4

turns out.) > > -ash > >> On 9 Aug 2018, at 19:22, Bolke de Bruin wrote: >> >> 0.5?? Can we score fractions :-) ? Sorry I missed this Ash. I think Fokko >> really wants a 1.10.1 quickly so better include it then? Can you make your >> vote +1? >>

Re: apache-airflow v1.10.0 on PyPi?

Hi Kris, Please use the mailing list for these kind of questions. Airflow 1.10.0 hasn’t been released yet. We are going through the motions, but it will take a couple of days before it’s official (if all goes well). B. Verstuurd vanaf mijn iPad > Op 9 aug. 2018 om 23:33 heeft Krish Sigler he

Re: [VOTE] Airflow 1.10.0rc4

I'd guess? > > -ash > >> On 8 Aug 2018, at 19:21, Bolke de Bruin wrote: >> >> Hey all, >> >> I have cut Airflow 1.10.0 RC4. This email is calling a vote on the release, >> which will last for 72 hours. Consider this my (binding) +1. >>

Re: Identifying delay between schedule & run instances

Hi vardang, What do you intent to gain from this metric? There are many influences that influence a difference between execution date and start date. You named one of them, but there are also functional ones (limits reached etc). We are not a real time system so we never really purposefully aim

Re: Plan to change type of dag_id from String to Number?

No we don’t have such plan. Dag ids are used to have a readable identifier. If you think it makes such a big difference in speed please show us numbers from Airflow running with and without. Thx B. Verstuurd vanaf mijn iPad > Op 6 aug. 2018 om 08:31 heeft vardangupta...@gmail.com > het volge

Re: [VOTE] Airflow 1.10.0rc4

;kubernetes'] > [2018-08-08 21:25:59,060] {cli.py:492} INFO - Running example_bash_operator.runme_0 2017-07-01T00:00:00+00:00 [None]> on host > 31ec1d1554b7 > > https://github.com/Fokko/incubator-airflow/commits/v1-10-stable > > Still no hard guarantees that 3.7 will

[VOTE] Airflow 1.10.0rc4

Hey all, I have cut Airflow 1.10.0 RC4. This email is calling a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow 1.10.0 RC 4 is available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/

Re: [VOTE] Airflow 1.10.0rc3

or release because it hard blocks > upgrading. > > George > >> On Tue, Aug 7, 2018 at 7:58 AM Bolke de Bruin wrote: >> >> Done. When I roll rc4 it will be part of it. >> >> >>> On 7 Aug 2018, at 16:26, Naik Kaxil wrote: >>> >&g

Re: [VOTE] Airflow 1.10.0rc3

on alembic migration execution. I have captured the issue in JIRA: >> https://issues.apache.org/jira/browse/AIRFLOW-2870 >> >> I would consider this a critical blocker for release because it hard blocks >> upgrading. >> >> George >> >>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

new vs old UI at this point? >> >> For many that upgrade will require some work to set up authentication on >> the new UI. This is fairly well documented in FAB. >> >> Max >> >> On Mon, Jul 23, 2018 at 3:08 AM Bolke de Bruin wrote: >> >>>

Re: [VOTE] Airflow 1.10.0rc3

k as that provides env vars at runtime but not > installtime, as far as I can tell.) > > >On Sun, Aug 5, 2018 at 5:20 PM Bolke de Bruin wrote: > >> +1 :-) >> >> Sent from my iPhone >> >>> On 5 Aug 2018, at 23:08, Ash Berlin-Taylor < >>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

After the release of 1.10 we will start removing the old “www”. B. > On 7 Aug 2018, at 11:03, Verdan Mahmood wrote: > > Hi Dave, > > AceJS is being used only in "www" version of webserver. > "www_rbac" does not use AceJS, and we implemented npm and webpack only for > www_rbac. > > Best, > *Ve

Re: Deploy Airflow on Kubernetes using Airflow Operator

Really awesome stuff. We are in progress to move over to k8s for Airflow (on prem though) and this is really helpful. B. Verstuurd vanaf mijn iPad > Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman > het volgende geschreven: > > Hi > > We at Google just open-sourced a Kubernetes custom contr

Re: [VOTE] Airflow 1.10.0rc3

> -ash > >> On 5 Aug 2018, at 22:05, Bolke de Bruin wrote: >> >> Yeah I figured it out. Originally i was using a different implementation of >> UTCDateTime, but that was unmaintained. I switched, but this version changed >> or has a different contract. Whi

Re: [VOTE] Airflow 1.10.0rc3

mestamp with time zone; > timestamptz > > 2018-08-03 00:00:00+00 > (1 row) > > postgres=# select '2018-08-03 01:00:00+01'::timestamp with time zone; > timestamptz > > 2018-08-03 00:00:00+00 > (1 row) >

Re: [VOTE] Airflow 1.10.0rc3

nge to log_filename_template from UPDATING.md in my other > branch) > > Setting the timezone to UTC per connection means the behaviour of Airflow > doesn't change depending on how the server is configured. > > -ash > >> On 5 Aug 2018, at 20:58

Re: [VOTE] Airflow 1.10.0rc3

Digging in a bit further. ti.dag_id / ti.task_id / ts / try_number .log is the format ts = execution_date.isoformat and should be in UTC afaik. something is weird tbh. B. > On 5 Aug 2018, at 21:32, Bolke de Bruin wrote: > > Ash, > > Readin

Re: [VOTE] Airflow 1.10.0rc3

>> <https://github.com/apache/incubator-airflow/pull/3112> >>> >>> I see two options for this - either conditionally enable k8s:// support if >>> the import works, or (less preferred) add kube-client to the emr deps >>> (which I like less) >>&

Re: [VOTE] Airflow 1.10.0rc3

us we transform it to UtC. I'll have a look at what needs to be done. Conditional import sounds fine to me. B. Sent from my iPhone > On 5 Aug 2018, at 19:14, Ash Berlin-Taylor wrote: > > >> On 5 Aug 2018, at 18:01, Bolke de Bruin wrote: >> >> Hi Ash, >

Re: [VOTE] Airflow 1.10.0rc3

ttps://github.com/apache/incubator-airflow/pull/3112 >>> <https://github.com/apache/incubator-airflow/pull/3112> >>> >>> I see two options for this - either conditionally enable k8s:// support if >>> the import works, or (less preferred) add kube-client to the emr dep

Re: [VOTE] Airflow 1.10.0rc3

rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz > > Cheers, Fokko > > 2018-08-03 9:47 GMT+02:00 Bolke de Bruin : > >> Hey all, >> >> I have cut Airflow 1.10.0 RC3. This email is calling a vote on the release, >> which will last for 72 hours. Consider this my

Re: The need for LocalTaskJob

It is actually all Executors doing this (at least Local, Celery). And yes (although you description is a bit cryptic) I think you are right. What you call “shelling out” does not really cover what happens on a process level though. We execute “bash -c” with “shell=True” which probably makes the

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

Here you go: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals B. Verstuurd vanaf mijn iPad > Op 31 jul. 2018 om 06:49 heeft Tao Feng het volgende > geschreven: > > Hey Jakob, > > Are we going to start AIF for airflow? Currently it is a little bit hard to >

Re: Kerberos and Airflow

rly on in this thread (passing keys from the > executor to workers). > > Happy to set up some time to talk real-time about this by the way, once we > iron out the details I want to implement whatever the best solution we come > up with is. > >> On Thu, Aug 2, 2018 at 4

The need for LocalTaskJob

Hi Max, Dan et al, Currently, when a scheduled task runs this happens in three steps: 1. Worker 2. LocalTaskJob 3. Raw task instance It uses (by default) 5 (!) different processes: 1. Worker 2. Bash + Airflow 3. Bash + Airflow I think we can merge worker and LocalTaskJob as the latter seem

[VOTE] Airflow 1.10.0rc3

Hey all, I have cut Airflow 1.10.0 RC3. This email is calling a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow 1.10.0 RC 3 is available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/

Re: Kerberos and Airflow

w would you make sure that other users > did not create DAGs that matched this regex? > > On Thu, Aug 2, 2018 at 1:51 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > >> Hi Dan, >> >> I discussed this a little bit with one of the security architect

Re: Kerberos and Airflow

Also: using the Kubernetes executor combined with some of the things we discussed greatly enhances the security of Airflow as the environment isn’t really shared anymore. B. > On 2 Aug 2018, at 19:51, Bolke de Bruin wrote: > > Hi Dan, > > I discussed this a little bit

Re: Kerberos and Airflow

lient might be > serializing arguments for hooks/operators that have been deprecated in > newer versions of the hooks, or the serialized DAG schema changes and old > DAGs aren't automatically updated. Might want to have some kind of > versioning system for serialized DAGs

Re: Kerberos and Airflow

if there is a secret/keytab that can be read by DAG A generated by > file X, and file X generates DAG B as well, there needs to be a scheme to > stop the parsing of DAG B on the scheduler from being able to read the > secret in DAG A. > > Does that make sense? > > On S

Re: Kerberos and Airflow

beros, curious what your thoughts are there. How would secrets be passed > securely in a multi-tenant Scheduler starting from parsing the DAGs up to > the executor sending them off? > > On Sat, Jul 28, 2018 at 5:07 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > >

Re: Kerberos and Airflow

ed the hook and operator to use “conn_id” instead of the non standard http_conn_id. Makes sense? B. * The BaseHook is adjusted to not connect to the database > On 28 Jul 2018, at 17:50, Bolke de Bruin wrote: > > Well, I don’t think a hook (or task) should be obtain it by itself.

Review on GPL fix required for 1.10 release

Hi Committers et al, Can I get a review on https://github.com/apache/incubator-airflow/pull/3660 it is required to be able to restart the voting process for 1.10. Thx Bolke

Re: Kerberos and Airflow

ng a customized > CredentialsReader class. Although hooks are technically private, I think a > lot of companies depend on them so the PMC should probably discuss if this > is an Airflow 2.0 change or not. > > On Fri, Jul 27, 2018 at 5:24 PM Bolke de Bruin wrote: > >

[REVOKE][VOTE] Release Airflow 1.10.0

FYI, Revoking the vote to address (mostly) license issues. Cheers Bolke > Begin forwarded message: > > From: Bolke de Bruin > Subject: Re: [VOTE] Release Airflow 1.10.0 > Date: 28 July 2018 at 12:52:28 CEST > To: gene...@incubator.apache.org > > Hi Justin, > >

Re: Kerberos and Airflow

Small correction: we probably need something like this https://github.com/tuxberlin/python-keyctl For proper securing Verstuurd vanaf mijn iPad > Op 27 jul. 2018 om 23:24 heeft Bolke de Bruin het > volgende geschreven: > > Sure. In general I consider keytabs as a part o

Re: Kerberos and Airflow

lgende geschreven: > > I'm curious if you had any ideas in terms of ideas to enable multi-tenancy > with respect to Kerberos in Airflow. > >> On Fri, Jul 27, 2018 at 2:38 PM Bolke de Bruin wrote: >> >> Cool. The doc will need some refinement as it isn't entirel

Re: Kerberos and Airflow

ttps://blog.tedmiston.com/> | CV >> <https://stackoverflow.com/cv/taylor> | LinkedIn >> <https://www.linkedin.com/in/tedmiston/> | AngelList >> <https://angel.co/taylor> | Stack Overflow >> <https://stackoverflow.com/users/149428/taylor-edmiston> >&

Re: Sep Airflow Bay Area Meetup @ Google

Great stuff Feng Lu! Sent from my iPhone > On 24 Jul 2018, at 21:49, Feng Lu wrote: > > The meetup event is now available for people to sign up: > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/253105418/. > yay! > > On Mon, Jul 23, 2018 at 3:40 PM Chris Riccomini > w

  1   2   3   4   5   6   7   8   >