Re: Deprecate legacy UI in favor of FAB RBAC

The branch has for 1.10 is different from master (v1-10-test and stable) so there is no issue from that perspective. If there are fixes for 1.10 after the merge that concern the old ui they can just be based against the right branch. B. Op di 18 dec. 2018 13:49 schreef airflowuser I don't

Re: [RESULT] Graduate Apache Airflow as a TLP

an, Kevin, Marc, Sunil, > > >> > Adityan, Deng, Neelesh, Sai > > >> > > > >> > I'll use this result to start the corresponding VOTE on the IPMC. > I'm > > >> > at an offsite today, so I have limited email time.

Re: [VOTE] Graduate the Apache Airflow as a TLP

workflow automation and scheduling that can be >>>> used to author and manage data pipelines; and be it further >>>> >>>> RESOLVED, that the office of "Vice President, Apache Airflow" be >>>> and hereby is created, the person hold

Re: [VOTE] Graduate the Apache Airflow as a TLP

Directors as the chair >> of the Apache Airflow Project, and to have primary responsibility >> for management of the projects within the scope of >> responsibility of the Apache Airflow Project; and be it further >> >> RESOLVED, that the persons listed immediately

Re: [DISCUSS] Apache Airflow graduation from the incubator

Ping! Verstuurd vanaf mijn iPad > Op 27 nov. 2018 om 21:39 heeft Bolke de Bruin het > volgende geschreven: > > Hi Folks, > > Thanks all for your responses and particularly Stefan for his suggestion to > use the generic Apache way to handle security issues. This seems

Re: [DISCUSS] Apache Airflow graduation from the incubator

ar, but the project did two >>> releases during that time, once TLP releasing becomes easier :) >>> >>> Regarding QU30 you may consider to use the ASF wide security mailing >>> list [3] and process [4]. >>> >>> Kind Regards, >>>

Re: [DISCUSS] Apache Airflow graduation from the incubator

d two >> releases during that time, once TLP releasing becomes easier :) >> >> Regarding QU30 you may consider to use the ASF wide security mailing >> list [3] and process [4]. >> >> Kind Regards, >> Stefan >> >> [3] https://www.apache.org/se

Re: [DISCUSS] Apache Airflow graduation from the incubator

Ping! Sent from my iPhone > On 24 Nov 2018, at 12:57, Bolke de Bruin wrote: > > Hi All, > > With the Apache Airflow community healthy and growing, I think now would be a > good time to > discuss where we stand regarding to graduation from the Incubator, and what &

[DISCUSS] Apache Airflow graduation from the incubator

Hi All, With the Apache Airflow community healthy and growing, I think now would be a good time to discuss where we stand regarding to graduation from the Incubator, and what requirements remains. Apache Airflow entered incubation around 2 years ago, since then, the Airflow community

Re: Remove airflow from pypi

Agree! This is even a security issue. Sent from my iPhone > On 23 Nov 2018, at 15:29, Driesprong, Fokko wrote: > > Hi all, > > I think we should remove airflow (not > apache-airflow) from Pypi. I still get questions from people who > accidentally install

Re: Airflow 1.10.1 is released

That was pretty smooth sailing! Well done! Sent from my iPhone > On 22 Nov 2018, at 04:13, Kevin Yang wrote: > >  Thank you Ash for cutting the release! > >> On Wed, Nov 21, 2018 at 6:09 PM Sid Anand wrote: >> >> Excellent work Ash! Thanks for doing the needful!! >> >> -s >> >>> On Wed,

Re: [VOTE CANCELED] Airflow 1.10.1rc1

You need to wait 72h and until you have 3+ votes. Satisfy both conditions :-) On putting things on pypi I'm personally in favor. It's not an Apache channel thus not official. On the other hand Apache is Java focused and only understands SNAPSHOTS but that's essentially the same as this. B.

Re: Airflow 1.10.1b1 release available - PLEASE TEST

Please make sure not to call it release at least  Sent from my iPhone > On 12 Nov 2018, at 22:51, Ash Berlin-Taylor wrote: > > Hi Hitesh, > > My understanding was that the only official place an Apache release can > happen from is https://www.apache.org/dist/incubator/airflow/ >

Re: 1.10.1 Release?

The fix is in master and should work across all DST changes. It will be included in 1.10.1. B. Sent from my iPhone > On 5 Nov 2018, at 19:54, Dave Fisher wrote: > > > >> On 2018/10/28 00:09:05, Bolke de Bruin wrote: >> I wonder how to treat this: >> &

Re: 1.10.1 Release?

They are :-) I might have to adjust some tests, let's see. Will do that tomorrow if required. B. > On 30 Oct 2018, at 21:53, Ash Berlin-Taylor wrote: > > Fair :) Timezones are _hard_ > > Giving it a look now. > > -ash > >> On 30 Oct 2018, at 20:50, Bolke de B

Re: 1.10.1 Release?

> Ash > > On 30 October 2018 19:40:27 GMT, Bolke de Bruin wrote: > I think we should use the UTC date for cron instead of the naive local date > time. I will check of croniter implements this so we can rely on that. > > B. > > On 28 Oct 2018, at 02:09

Re: 1.10.1 Release?

Patch available at: https://github.com/apache/incubator-airflow/pull/4117 please test. B. > On 30 Oct 2018, at 21:14, Bolke de Bruin wrote: > > We should just pass it the UTC date (we should never use local time except at > the user interface). I’m testing a patch right

Re: 1.10.1 Release?

ll give 2:55 CEST followed by 2:00 CET) and then we convert it to > UTC for return - but right now we are giving it a TZ-unaware local time. > > I think. > > Ash > > On 30 October 2018 19:40:27 GMT, Bolke de Bruin wrote: > I think we should use the UTC date for cron ins

Re: 1.10.1 Release?

I think we should use the UTC date for cron instead of the naive local date time. I will check of croniter implements this so we can rely on that. B. > On 28 Oct 2018, at 02:09, Bolke de Bruin wrote: > > I wonder how to treat this: > > This is what I think happens (need to ver

Re: 1.10.1 Release?

nblock the scheduler. > > Ash > > On 30 October 2018 18:43:30 GMT, David Klosowski wrote: >> Hi Airflow Devs: >> >> Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1? >> >> Thanks. >> >> Regards, >> David >> >&

Re: 1.10.1 Release?

ze(datetime(2017, 3, 26)) >>>> val = croniter('0 0 * * *', local_date).get_next(datetime) > > I think the problem is that we are _not_ passing a TZ aware dag in and we > should be. > >> On 30 Oct 2018, at 17:35, Bolke de Bruin wrote: >> >> Oh that’s

Re: 1.10.1 Release?

>>>> local_date = tz.localize(datetime(2017, 3, 26)) >>>> val = croniter('0 0 * * *', local_date).get_next(datetime) > > I think the problem is that we are _not_ passing a TZ aware dag in and we > should be. > >> On 30 Oct 2018, at 17:35, Bolke de Bru

Re: 1.10.1 Release?

>> to parse the dag... and it never returns. >> >> I've only just managed to reproduce it, so haven't dug in to why yet. A >> quick hacky debug print shows something is stuck in an infinite loop. >> >> -ash >> >>> On 29 Oct 2018, at 17:5

Re: explicit_defaults_for_timestamp for mysql

changes have been working for us so far. > > On Tue, Oct 23, 2018 at 10:09 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > We only need it at table creation time or alter table time during which an > alembic script would fail if MySQL restarts I assume? > > I'm not

Re: 1.10.1 Release?

it. > > Bolke: are there any outstanding tasks/issues that you know of that might > slow down the vote for a 1.10.1? (i.e. did we sort of out all the licensing > issues that were asked of us? I thought I read something about license > declarations in markdown files?) &

Re: 1.10.1 Release?

> > > ‐‐‐ Original Message ‐‐‐ > On Friday, October 26, 2018 6:12 PM, Bolke de Bruin wrote: > >> Hey Ash, >> >> I was wondering if you are picking up the 1.10.1 release? Master is speeding >> ahead and you were tracking fixes for 1.10.1 right? >> >> B. > >

1.10.1 Release?

Hey Ash, I was wondering if you are picking up the 1.10.1 release? Master is speeding ahead and you were tracking fixes for 1.10.1 right? B.

Re: explicit_defaults_for_timestamp for mysql

O and reading https://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_explicit_defaults_for_timestamp indicates that it can be set on the session level as well. So

Re: execution_date - can we stop the confusion?

I dont think this makes sense and I dont that think anyone had a real issue with this. Execution date has been clearly documented and is part of the core principles of airflow. Renaming will create more confusion. Please note that I do think that as an anonymous user you cannot speak for any

Re: Airflow: Apache Graduation

where the Board approves it. Project is officially TLP once that is > done. > 5) Champion cleans up incubator resources, creates new TLP resources. > Example: https://jira.apache.org/jira/browse/INFRA-10639 > > -Jakob > > > On 21 September 2018 at 13:49, Bolke de Bruin

Re: Airflow: Apache Graduation

There is a maturity check that we can do. Don't have the link ready now. But can look it up later. To me two things need to be more mature 1. Handling of security issues 2. Licensing. 2a having ppl aware that with new dependencies updates to dependencies the respective licenses need to be

Re: Sep Airflow Bay Area Meetup @ Google

gt; > On Fri, Sep 14, 2018 at 11:42 PM Feng Lu wrote: > >> Not going to happen for this time, we don't receive enough interest from >> the community. >> >> >> On Wed, Sep 12, 2018 at 7:57 AM Bolke de Bruin wrote: >> >>> btw how are we doing

Re: Database referral integrity

Adding these kind of checks which work for integrity well make database access pretty slow. In addition it isnt there because in the past there was no strong connection between for example tasks and dagruns, it was more or less just coincidental. There also so some bisecting tools that probably

Live Airflow Meetup @ Amsterdam

https://www.youtube.com/watch?v=MM8tfTrcnfk Enjoy! Bolke

Re: Sep Airflow Bay Area Meetup @ Google

btw how are we doing on the “one day” hackathon? > On 12 Sep 2018, at 16:49, Bolke de Bruin wrote: > > Hi feng, > > I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam > (that I did together with Fokko). > > Cheers > Bolke > >&

Re: Sep Airflow Bay Area Meetup @ Google

Hi feng, I can do “Elegant pipelining with Airflow” recycle of pydata 2018 amsterdam (that I did together with Fokko). Cheers Bolke > On 4 Sep 2018, at 22:13, Feng Lu wrote: > > We are 3 weeks away from the meetup and still have a few lightening talks > open, please take the chance and share

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

> On Mon, Sep 10, 2018 at 2:10 PM, Bolke de Bruin wrote: > >> I cannot connect this to timezone issues yet. The trigger dagrun operator >> seems to work fine (checked the examples). >> >> B. >> >>> On 10 Sep 2018, at 08:19, Deng Xiaodong wrote

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

I cannot connect this to timezone issues yet. The trigger dagrun operator seems to work fine (checked the examples). B. > On 10 Sep 2018, at 08:19, Deng Xiaodong wrote: > > Hi Goutam, > > Seems it’s due to the timezone setting? > > If that’s the case, you may try either adjusting your

Re: Call for fixes for Airflow 1.10.1

ncubator-airflow/pull/3693 > > > > > > This is a bug in terms of metadata, which also affects the UI > > “Browse->Jobs”. > > > > > > Thanks. > > > > > > Regards, > > > > XD > > > > On Wed, Sep 5, 2018 at 23:55 B

Re: [VOTE][RESULT] Replace with Gitter with Slack?

; >- Sid Anand > > 0: 2 votes > >- Bolke de Bruin >- Kaxil Naik > > -0.5: 1 vote > >- Arthur Wiedmer > > > Vote result is a net positive of +0.5. > > I counted all of the PMC/committers' votes as binding. > > -s > > On Sat, Se

Re: [VOTE][RESULT] Replace with Gitter with Slack?

t;- Pedro Machado >- Scott Halgrim >- Eamon Keane >- Adam Boscarino >- Daniel Cohen >- Chandu Kavar >- William Horton >- Ben Gregory > > 0: 2 votes > >- Bolke de Bruin >- Kaxil Naik > > > -0.5: 1 vote > >-

Re: TriggerDagRunOperator sub tasks are scheduled to run after few hours

Can you share the definition? And timezone settings? Op vr 7 sep. 2018 20:53 schreef Maxime Beauchemin < maximebeauche...@gmail.com>: > Is the issue timezone related? Personally I've only used Airflow in > UTC-aligned environments so I can't help much on this topic. Bolke as > contributed

Re: Security issue being ignored?

Both are not security vulnerabilities: either it is in an upstream project or it is due to the way Airflow can be used. PR is welcome for the second JIRA. B. Verstuurd vanaf mijn iPad > Op 6 sep. 2018 om 11:07 heeft airflowuser > het volgende geschreven: > > Another example: >

Re: [VOTE] Replace with Gitter with Slack?

0 (binding) abstaining from the vote. I am not convinced we gain anything except disruption for an existing community. We cannot even force people to move away only make it “less” official. My question would be do we get more attendance of contributors and committers if we move to slack?

Re: Call for fixes for Airflow 1.10.1

You should push these to v1-10-test not to stable. Only once we start cutting RCs you should push to -stable. See the docs. This ensures a stable “stable”branch. Cheers Bolke. > On 3 Sep 2018, at 14:20, Ash Berlin-Taylor wrote: > > Hi everyone, > > I'm starting the process of gathering

Re: Add git tag for 1.10

I’m not sure what you think is off? The change is only in master not in 1.10. If you are reading the docs for updating to master then you are using the wrong docs if you are upgrading to 1.10. B. Verstuurd vanaf mijn iPad > Op 3 sep. 2018 om 14:53 heeft Robin Edwards het volgende >

Re: Running unit tests against SLUGIFY_USES_TEXT_UNIDECODE and AIRFLOW_GPL_UNIDECODE (also is this broken?)

The dependency is so small, besides python-slugify executes tests with both that I wouldn’t bother. Probably it makes more sense to get rid of python-nvd3 and to be able to upgrade d3. Verstuurd vanaf mijn iPad > Op 28 aug. 2018 om 22:18 heeft Taylor Edmiston het > volgende geschreven: >

Re: PR Review Dashboard?

I’m not in favor of moving to GitHub issues. While JIRA is not perfect it actually moves discussion to the mailing list. With GitHub issues the stuff just gets lost imho. I do like our changelogs a lot better now. B. Verstuurd vanaf mijn iPad > Op 27 aug. 2018 om 05:53 heeft Holden Karau

Re: Python 3.6 Support for Airflow 1.10.0

Let’s not drop 2.7 too quickly but maybe mark it deprecated. I’m pretty sure Airbnb still runs on 2.7. Also RedHat does not deliver python 3 in its enterprise edition yet by default so it will put enterprise users in a bit of an awkward spot. B. Verstuurd vanaf mijn iPad > Op 28 aug. 2018 om

Re: Cloudera Hue in License

Yes we do. The kerberos ticket renewer i amended from hue. Sent from my iPhone > On 24 Aug 2018, at 17:46, Taylor Edmiston wrote: > > For anyone else who's curious - it looks like Hue is a web app for Hadoop. > > https://github.com/cloudera/hue > > https://en.wikipedia.org/wiki/Hue_(Hadoop)

Re: [RESULT][VOTE] Release Airflow 1.10.0

; wrote: >>>> >>>>> Thanks Bolke! >>>>> >>>>> I've just pushed the artifacts to Apache Dist: >>>>> >>>>> https://dist.apache.org/repos/dist/release/incubator/ >>>> airflow/1.10.0-incubating/ >>&

Fwd: [RESULT][VOTE] Release Airflow 1.10.0

iPhone Begin forwarded message: > From: Bolke de Bruin > Date: 20 August 2018 at 20:00:28 CEST > To: gene...@incubator.apache.org, dev@airflow.incubator.apache.org > Subject: [RESULT][VOTE] Release Airflow 1.10.0 > > The vote to release Airflow 1.10.0-incubating, having been

[RESULT][VOTE] Release Airflow 1.10.0

message: > From: Bolke de Bruin > Date: 20 August 2018 at 19:56:23 CEST > To: gene...@incubator.apache.org > Subject: Re: [VOTE] Release Airflow 1.10.0 (new vote based on rc4) > > Appreciated Hitesh. Do you know how to add headers to .MD files? There seems > to be no tec

Fwd: [VOTE] Release Airflow 1.10.0 (new vote based on rc4)

Hi Folks, If you have contacts with IPMC members that can vote binding, can you please reach out to them and ask for help? The vote is stalled at the moment and without 3 +1's we can't release Airflow 1.10.0 Cheers Bolke -- Forwarded message -- From: Bolke de Bruin Date: 2018

Re: apache-airflow v1.10.0 on PyPi?

> | LinkedIn >> <https://www.linkedin.com/in/tedmiston/> | AngelList >> <https://angel.co/taylor> | Stack Overflow >> <https://stackoverflow.com/users/149428/taylor-edmiston> >> >> >> On Thu, Aug 9, 2018 at 5:43 PM, Krish Sigler wrote:

[RESULT][VOTE] Airflow 1.10.0rc4

Hello, Apache Airflow (incubating) 1.10.0 (based on RC4) has been accepted. 4 “+1” binding votes received: - Ash Berlin-Taylor (binding) - Naik Kaxil (binding) - Bolke de Bruin (binding) - Fokko Driesprong (binding) 1 “-1” non-binding: - Daniel Imberman (non-binding) My next step is to open

Re: [VOTE] Airflow 1.10.0rc4

turns out.) > > -ash > >> On 9 Aug 2018, at 19:22, Bolke de Bruin wrote: >> >> 0.5?? Can we score fractions :-) ? Sorry I missed this Ash. I think Fokko >> really wants a 1.10.1 quickly so better include it then? Can you make your >> vote +1? >> >>

Re: apache-airflow v1.10.0 on PyPi?

Hi Kris, Please use the mailing list for these kind of questions. Airflow 1.10.0 hasn’t been released yet. We are going through the motions, but it will take a couple of days before it’s official (if all goes well). B. Verstuurd vanaf mijn iPad > Op 9 aug. 2018 om 23:33 heeft Krish Sigler

Re: [VOTE] Airflow 1.10.0rc4

? > > -ash > >> On 8 Aug 2018, at 19:21, Bolke de Bruin wrote: >> >> Hey all, >> >> I have cut Airflow 1.10.0 RC4. This email is calling a vote on the release, >> which will last for 72 hours. Consider this my (binding) +1. >> >> Airfl

Re: Identifying delay between schedule & run instances

Hi vardang, What do you intent to gain from this metric? There are many influences that influence a difference between execution date and start date. You named one of them, but there are also functional ones (limits reached etc). We are not a real time system so we never really purposefully

Re: Plan to change type of dag_id from String to Number?

No we don’t have such plan. Dag ids are used to have a readable identifier. If you think it makes such a big difference in speed please show us numbers from Airflow running with and without. Thx B. Verstuurd vanaf mijn iPad > Op 6 aug. 2018 om 08:31 heeft vardangupta...@gmail.com > het

Re: [VOTE] Airflow 1.10.0rc4

ng example_bash_operator.runme_0 2017-07-01T00:00:00+00:00 [None]> on host > 31ec1d1554b7 > > https://github.com/Fokko/incubator-airflow/commits/v1-10-stable > > Still no hard guarantees that 3.7 will be fully supported, but at least it > runs :-) > > Cheers, Fokko

[VOTE] Airflow 1.10.0rc4

Hey all, I have cut Airflow 1.10.0 RC4. This email is calling a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow 1.10.0 RC 4 is available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/

Re: [VOTE] Airflow 1.10.0rc3

r for release because it hard blocks > upgrading. > > George > >> On Tue, Aug 7, 2018 at 7:58 AM Bolke de Bruin wrote: >> >> Done. When I roll rc4 it will be part of it. >> >> >>> On 7 Aug 2018, at 16:26, Naik Kaxil wrote: >>> >&g

Re: [VOTE] Airflow 1.10.0rc3

or on alembic migration execution. I have captured the issue in JIRA: >> https://issues.apache.org/jira/browse/AIRFLOW-2870 >> >> I would consider this a critical blocker for release because it hard blocks >> upgrading. >> >> George >> &g

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

new vs old UI at this point? >> >> For many that upgrade will require some work to set up authentication on >> the new UI. This is fairly well documented in FAB. >> >> Max >> >> On Mon, Jul 23, 2018 at 3:08 AM Bolke de Bruin wrote: >> >>> I thi

Re: [VOTE] Airflow 1.10.0rc3

k as that provides env vars at runtime but not > installtime, as far as I can tell.) > > >On Sun, Aug 5, 2018 at 5:20 PM Bolke de Bruin wrote: > >> +1 :-) >> >> Sent from my iPhone >> >>> On 5 Aug 2018, at 23:08, Ash Berlin-Taylor < >>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

After the release of 1.10 we will start removing the old “www”. B. > On 7 Aug 2018, at 11:03, Verdan Mahmood wrote: > > Hi Dave, > > AceJS is being used only in "www" version of webserver. > "www_rbac" does not use AceJS, and we implemented npm and webpack only for > www_rbac. > > Best, >

Re: Deploy Airflow on Kubernetes using Airflow Operator

Really awesome stuff. We are in progress to move over to k8s for Airflow (on prem though) and this is really helpful. B. Verstuurd vanaf mijn iPad > Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman > het volgende geschreven: > > Hi > > We at Google just open-sourced a Kubernetes custom

Re: [VOTE] Airflow 1.10.0rc3

> -ash > >> On 5 Aug 2018, at 22:05, Bolke de Bruin wrote: >> >> Yeah I figured it out. Originally i was using a different implementation of >> UTCDateTime, but that was unmaintained. I switched, but this version changed >> or has a different contract. Whi

Re: [VOTE] Airflow 1.10.0rc3

> 2018-08-03 00:00:00+00 > (1 row) > > postgres=# select '2018-08-03 01:00:00+01'::timestamp with time zone; > timestamptz > > 2018-08-03 00:00:00+00 > (1 row) > > > > > -ash > >> On 5 Aug 2018, at 21:2

Re: [VOTE] Airflow 1.10.0rc3

name_template from UPDATING.md in my other > branch) > > Setting the timezone to UTC per connection means the behaviour of Airflow > doesn't change depending on how the server is configured. > > -ash > >> On 5 Aug 2018, at 20:58, Bolke de Bruin wr

Re: [VOTE] Airflow 1.10.0rc3

Digging in a bit further. ti.dag_id / ti.task_id / ts / try_number .log is the format ts = execution_date.isoformat and should be in UTC afaik. something is weird tbh. B. > On 5 Aug 2018, at 21:32, Bolke de Bruin wrote: > > Ash, > > Readin

Re: [VOTE] Airflow 1.10.0rc3

irflow/pull/3112> >>> >>> I see two options for this - either conditionally enable k8s:// support if >>> the import works, or (less preferred) add kube-client to the emr deps >>> (which I like less) >>> >>> Sorry - this is the first time I

Re: [VOTE] Airflow 1.10.0rc3

transform it to UtC. I'll have a look at what needs to be done. Conditional import sounds fine to me. B. Sent from my iPhone > On 5 Aug 2018, at 19:14, Ash Berlin-Taylor wrote: > > >> On 5 Aug 2018, at 18:01, Bolke de Bruin wrote: >> >> Hi Ash, >> >

Re: [VOTE] Airflow 1.10.0rc3

12 >>> <https://github.com/apache/incubator-airflow/pull/3112> >>> >>> I see two options for this - either conditionally enable k8s:// support if >>> the import works, or (less preferred) add kube-client to the emr deps >>> (which I like less) >>>

Re: [VOTE] Airflow 1.10.0rc3

rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz > > Cheers, Fokko > > 2018-08-03 9:47 GMT+02:00 Bolke de Bruin : > >> Hey all, >> >> I have cut Airflow 1.10.0 RC3. This email is calling a vote on the release, >> which will last for 72 hours. Consider this

Re: The need for LocalTaskJob

It is actually all Executors doing this (at least Local, Celery). And yes (although you description is a bit cryptic) I think you are right. What you call “shelling out” does not really cover what happens on a process level though. We execute “bash -c” with “shell=True” which probably makes

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

Here you go: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals B. Verstuurd vanaf mijn iPad > Op 31 jul. 2018 om 06:49 heeft Tao Feng het volgende > geschreven: > > Hey Jakob, > > Are we going to start AIF for airflow? Currently it is a little bit hard to >

Re: Kerberos and Airflow

d (passing keys from the > executor to workers). > > Happy to set up some time to talk real-time about this by the way, once we > iron out the details I want to implement whatever the best solution we come > up with is. > >> On Thu, Aug 2, 2018 at 4:13 PM Bolke de B

The need for LocalTaskJob

Hi Max, Dan et al, Currently, when a scheduled task runs this happens in three steps: 1. Worker 2. LocalTaskJob 3. Raw task instance It uses (by default) 5 (!) different processes: 1. Worker 2. Bash + Airflow 3. Bash + Airflow I think we can merge worker and LocalTaskJob as the latter

[VOTE] Airflow 1.10.0rc3

Hey all, I have cut Airflow 1.10.0 RC3. This email is calling a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow 1.10.0 RC 3 is available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/

Re: Kerberos and Airflow

w would you make sure that other users > did not create DAGs that matched this regex? > > On Thu, Aug 2, 2018 at 1:51 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > >> Hi Dan, >> >> I discussed this a little bit with one of the security architects her

Re: Kerberos and Airflow

Also: using the Kubernetes executor combined with some of the things we discussed greatly enhances the security of Airflow as the environment isn’t really shared anymore. B. > On 2 Aug 2018, at 19:51, Bolke de Bruin wrote: > > Hi Dan, > > I discussed this a litt

Re: Kerberos and Airflow

ments for hooks/operators that have been deprecated in > newer versions of the hooks, or the serialized DAG schema changes and old > DAGs aren't automatically updated. Might want to have some kind of > versioning system for serialized DAGs to at least ensure that stored DAGs > are

Re: Kerberos and Airflow

if there is a secret/keytab that can be read by DAG A generated by > file X, and file X generates DAG B as well, there needs to be a scheme to > stop the parsing of DAG B on the scheduler from being able to read the > secret in DAG A. > > Does that make sense? > > On S

Re: Kerberos and Airflow

beros, curious what your thoughts are there. How would secrets be passed > securely in a multi-tenant Scheduler starting from parsing the DAGs up to > the executor sending them off? > > On Sat, Jul 28, 2018 at 5:07 PM Bolke de Bruin <mailto:bdbr...@gmail.com>> wrote: > >

Re: Kerberos and Airflow

justed the hook and operator to use “conn_id” instead of the non standard http_conn_id. Makes sense? B. * The BaseHook is adjusted to not connect to the database > On 28 Jul 2018, at 17:50, Bolke de Bruin wrote: > > Well, I don’t think a hook (or task) should be obtain it by it

Re: Kerberos and Airflow

customized > CredentialsReader class. Although hooks are technically private, I think a > lot of companies depend on them so the PMC should probably discuss if this > is an Airflow 2.0 change or not. > > On Fri, Jul 27, 2018 at 5:24 PM Bolke de Bruin wrote: > >>

[REVOKE][VOTE] Release Airflow 1.10.0

FYI, Revoking the vote to address (mostly) license issues. Cheers Bolke > Begin forwarded message: > > From: Bolke de Bruin > Subject: Re: [VOTE] Release Airflow 1.10.0 > Date: 28 July 2018 at 12:52:28 CEST > To: gene...@incubator.apache.org > > Hi Justin, >

Re: Kerberos and Airflow

lgende geschreven: > > I'm curious if you had any ideas in terms of ideas to enable multi-tenancy > with respect to Kerberos in Airflow. > >> On Fri, Jul 27, 2018 at 2:38 PM Bolke de Bruin wrote: >> >> Cool. The doc will need some refinement as it isn't entirely accurate

Re: Sep Airflow Bay Area Meetup @ Google

Great stuff Feng Lu! Sent from my iPhone > On 24 Jul 2018, at 21:49, Feng Lu wrote: > > The meetup event is now available for people to sign up: > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/253105418/. > yay! > > On Mon, Jul 23, 2018 at 3:40 PM Chris Riccomini >

Re: Airflow talk at Munich Data Engineering Meetup

Hi Stefan, Great stuff! Can you make sure to at least once use “Apache Airflow (incubating)” in the text, preferably at the beginning of the paragraph? That would be greatly appreciated. Thanks Bolke Verstuurd vanaf mijn iPad > Op 24 jul. 2018 om 07:54 heeft Stefan Seelmann het > volgende

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

I think it should be removed now. 1.10.X should be the last release seri s that supports the old www. Do we need to vote on this? Great work Verdan! Verstuurd vanaf mijn iPad > Op 23 jul. 2018 om 10:23 heeft Driesprong, Fokko het > volgende geschreven: > > ​Nice work Verdan. > > The

Re: [VOTE] Release Airflow 1.10.0

Hi Justin, Thank you for the thorough review! I have created AIRFLOW-2779 to track most of the issues you have raised. On the GPL dependency you mentioned. We are not distributing GPL sources, not in source or in binary form. This has never been the case. In the third degree there

Re: [VOTE] Release Airflow 1.10.0

It is just a convenience package and not part of the vote, but that argument seems mood. I will therefore remove the binary package. B. Verstuurd vanaf mijn iPad > Op 20 jul. 2018 om 16:58 heeft Bertrand Delacretaz > het volgende geschreven: > >> Le ven. 20 juil. 2018 à 16:15, sebb a écrit

[RESULT][VOTE] Airflow 1.10.0rc2

Hello, (Just making sure this works with the IPMC) Apache Airflow (incubating) 1.10.0 (based on RC2) has been accepted. 4 “+1” binding votes received: - Joy Gao (binding) - Naik Kaxil (binding) - Bolke de Bruin (binding) - Fokko Driesprong (binding) My next step is to open a thread

Re: [RESULT][VOTE] Airflow 1.10.0rc2

well my name is Bolke, but you get what I mean :P > On 18 Jul 2018, at 20:14, Bolke de Bruin wrote: > > Hello, > > Apache Airflow (incubating) 1.10.0 (based on RC2) has been accepted. > > 4 “+1” binding votes received: > > - Joy Gao (binding) > - Naik Kaxi

[RESULT][VOTE] Airflow 1.10.0rc2

Hello, Apache Airflow (incubating) 1.10.0 (based on RC2) has been accepted. 4 “+1” binding votes received: - Joy Gao (binding) - Naik Kaxil (binding) - Bolke de Bruin (binding) - Fokko Driesprong (binding) My next step is to open a thread with the IPMC. Cheers, Chris

Re: Airflow support for kubernetes Exceutor

Yes it Will. Sent from my iPhone > On 18 Jul 2018, at 10:19, ramandu...@gmail.com wrote: > > will 1.10 release have support for kubernetes Executor. > > Thanks, > Raman

  1   2   3   4   5   6   >