Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

ugly. >>>>>>>> >>>>>>>> On Mon, Jun 26, 2017 at 2:34 PM, Alex Guziel < >>>> alex.guz...@airbnb.com >>>>> . >>>>>>>> invalid >>>>>>>>> wrote: >>>>&g

Re: Airflow profiling

Free version also there, maybe more integration testing and benchmarking. https://stackimpact.com/pricing/ <https://stackimpact.com/pricing/> B. > On 27 Jun 2017, at 22:00, Chris Riccomini <criccom...@apache.org> wrote: > > Seems you have to pay? > > On Tue, Jun 27,

Airflow profiling

Just saw this tool on hacker news: https://github.com/stackimpact/stackimpact-python Might be interesting for some profiling. Bolke

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

We have been running it for the last couple of days. No issues and seems more responsive. +1, binding Bolke > On 25 Jun 2017, at 01:10, Maxime Beauchemin > wrote: > > Dear all, > > 1.8.2 RC2 is baked and available at: >

Re: Airflow 1.8.1 scheduler issue

This will be fixed in 1.8.2 which will be out shortly (rc2 has the fix). Bolke > On 22 Jun 2017, at 20:50, Drew Zoellner > wrote: > > Hi airflow dev team, > > We have a subdag which looks like the following... > > > > This subdag has a concurrency limit of

Re: Release Manager for 1.8.2?

ks alright before I send an announcement about > 1.8.2rc2 to the list? > > Max > > On Wed, Jun 21, 2017 at 10:44 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Please also include https://github.com/apache/incubator-airflow/pull/2022 >> >> I have ha

Re: Role Based Access Control for Airflow UI

One downside I see from FAB is that is does not do Business Role mapping to FAB role. I would prefer to create groups in IPA/LDAP/AD and have those map to FAB roles instead of needing to manage that in FAB. B. > On 22 Jun 2017, at 09:36, Bolke de Bruin <bdbr...@gmail.com> wrote: >

Re: Role Based Access Control for Airflow UI

>>> >>> I wouldn't be too worried about the permissions about refreshing DAGs, as >>> far as I know this functionality is no longer required with the new >>> webservers which reload state periodically, and will certainly be removed >>> when we hav

Re: Airflow Logging Improvements

In the light of fixing logging, I would definitely appreciate written design. Especially, as there have been multiple attempts to fix some issues but these have been more like stop gap fixes. In my opinion Airflow should not stipulate in a hard coded fashion where and how logging takes place.

Re: Release Manager for 1.8.2?

l of these, though if there are any merge > conflicts I'll pass and move their Fix-Version to 1.9.0. > > Max > >> On Wed, Jun 21, 2017 at 1:20 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: >> >> Hi max, >> >> PR 2365 is merged into master and also int

Re: airflow with MSSQL db

It is not officially supported but I know of some that run airflow on top of mssql. We do integrate fixes for it and did so in the past. The sql we run is not overly complex, so in general you should be fine. I would stress test it with a couple of examples and run the tests against it.

Re: Release Manager for 1.8.2?

e as far as I know for 1.8.2. Bolke P.S. A friendly request if any of the committers do +1 or a LGTM to a PR, please also merge the PR instead of leaving it to others. “airflow-pr merge XXX” is there to help. > On 16 Jun 2017, at 15:11, Bolke de Bruin <bdbr...@gmail.com> wrote: > > I

Re: Release Manager for 1.8.2?

Found the issue for the permission denied part: beeline is not executable in the latest tarball from Cloudera. I’ll have a workaround for this in a few minutes and will push it to Apache right away. Bolke. > On 14 Jun 2017, at 18:15, Bolke de Bruin <bdbr...@gmail.com> wrote: &g

Re: Airflow DAG deadlock, "SKIPPED" state not cascading

I have created PR https://github.com/apache/incubator-airflow/pull/2365 <https://github.com/apache/incubator-airflow/pull/2365> for this issue. Bolke > On 14 Jun 2017, at 16:26, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Sorry missed your comment on the dag. Will hav

Re: Airflow DAG deadlock, "SKIPPED" state not cascading

se/AIRFLOW-1296 a few days ago. It > includes a DAG I can consistently reproduce this with on 1.8.1 and master. > I get the "This should not happen" log message as well and the DAG fails. > > On Wed, Jun 14, 2017 at 3:27 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: &g

Re: Airflow DAG deadlock, "SKIPPED" state not cascading

CircuitOperator > and LatestOnlyOperator. I filed > https://issues.apache.org/jira/browse/AIRFLOW-1296 a few days ago. It > includes a DAG I can consistently reproduce this with on 1.8.1 and master. > I get the "This should not happen" log message as well and the DAG fails. > >> O

Re: Airflow DAG deadlock, "SKIPPED" state not cascading

Please provide the full logs (you are cutting out too much info), dag definition (sanitized), airflow version. Bolke Sent from my iPhone > On 13 Jun 2017, at 23:51, Rajesh Chamarthi wrote: > > I currently have a dag which follows the following pattern > >

1.8.2rc1 on pypi?

Hi Max, Did you accidentily push 1.8.2rc1 to pypi? That’s a little bit at odds with the Apache release process (although it is not an official channel). Bolke

Re: Role Based Access Control for Airflow UI

Will respond but im traveling at the moment. Give me a few days. Sent from my iPhone > On 12 Jun 2017, at 13:39, Chris Riccomini wrote: > > Hey all, > > Checking in on this. We spent a good chunk of time thinking about this, and > want to move forward with it, but want

Re: Tasks Queued but never run

; relevant JIRA Issue off hand? > > On Wed, Jun 7, 2017, 4:28 PM Bolke de Bruin <bdbr...@gmail.com> wrote: > >> It is 1.8.x specific in this case (for backfills). >> >> Sent from my iPhone >> >>> On 7 Jun 2017, at 21:35, Russell Pierce <russell.s.pie...@g

Re: Release Manager for 1.8.2?

Hi Max, Sounds good. Couple of things: * Can I suggest using the v1-8-test branch as the branch to be used for preparing the rc? If we hit RC then move it over to v1-8-stable? V1-8-test already had some fixes in that should land in 1.8.2 and the RC should be tagged in the stable branch. That

Re: Tasks Queued but never run

Jun 2017, at 12:04, Bolke de Bruin <bdbr...@gmail.com> wrote: > > I can confirm the issue (havent found the cause yet), but this is with > BACKFILLS which function independently > from the scheduler. So restarting the scheduler will not help. > > Bolke > >> On

Re: Tasks Queued but never run

I can confirm the issue (havent found the cause yet), but this is with BACKFILLS which function independently from the scheduler. So restarting the scheduler will not help. Bolke > On 6 Jun 2017, at 19:35, Noah Yetter wrote: > > I'm experiencing the same issue. I've built a

Re: task failure propagates correctly in sequential executor but not in celery executor

Parallel executing comes to mind. What version of Airflow are you running (always report this) and please provide full logs (processor, scheduler, worker). Thanks Bolke > On 7 Jun 2017, at 00:13, Ali Naqvi wrote: > > Hi folks, > > So it turns out in the

Re: Release Manager for 1.8.2?

. Cheers Bolke > On 18 May 2017, at 20:54, Bolke de Bruin <bdbr...@gmail.com> wrote: > > https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow > <https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow> > > (See higher up in the

Re: Cloud ML Operators

Hi Peter, That sounds great! I think the main criteria for this is will you maintain the code afterwards? The contrib section is slowly but steadily growing and with operators/hooks we are particularly dependent on the community as not all (or even none in some case) of the committers use

Re: Airflow HA @ING

> -- Hitesh > > [1] https://www.apache.org/foundation/marks/#books > <https://www.apache.org/foundation/marks/#books> > > On Tue, May 30, 2017 at 12:32 PM, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdbr...@gmail.com>> wrote: > Hi, > >

Airflow HA @ING

Hi, Just wanted to let you know that one of my team members, Johan Witman, has been writing up on how we are configuring Airflow in HA. We aren’t done yet and some patches will need to land in Airflow to make everything work, but it might inspire others to try it out so we can gather

Re: Concurrent schedulers

s communication to master process) > * Define a clear master cycle that would read the list of accumulated task > instances from the DB, dedup, prioritize and schedule. That master cycle > should have a lock (and lock expiration) as well. > > Max > > On Mon, May 22, 2017 at 12:27

Re: Concurrent webservers

You should be absolutely fine. Please note that you need to keep your DAG dirs in sync (more or less for the webserver). > On 22 May 2017, at 07:53, Stephen Rigney wrote: > > Hi, > > We're running airflow in production, but we'd like to confirm if it is safe > to spawn

Re: Concurrent schedulers

Hi Stephen, We are currently stress testing Airflow for use in a multi-master setup. One of my team members is doing a write up that should show up online shortly. TL;DR; in its current state Airflow will need some patches in order to run concurrently. One issue is that Airflow can have a

Re: Release Manager for 1.8.2?

set of > "cherries" that we use at Airbnb in production and every bugfix / minor > feature that looks benign to us. Given that, we're committing to try out RC > along with everyone else. > > What cadence are we aiming at? What should be the target date for the RC? > > Max

Re: Release Manager for 1.8.2?

gt; cherries we run in production already at Airbnb, holding the 50+ extra > commits in master for 1.8.3. This is marginally useful but helps getting > the release mechanics oiled up. > > I'm trying to be as transparent as I can here, and open to discuss the > different ways we can

Fwd: Release Manager for 1.8.2?

PING. > Begin forwarded message: > > From: Bolke de Bruin <bdbr...@gmail.com> > Subject: Release Manager for 1.8.2? > Date: 14 May 2017 at 13:44:39 GMT+2 > To: dev@airflow.incubator.apache.org > > Hi Folks, > > With 1.8.1 we have very much improved the

Re: Simple Airflow BashOperators run but can't be scheduled or un-paused

book.com/jurney> datasyndrome.com >> >> On Tue, May 16, 2017 at 2:18 PM, Russell Jurney <russell.jur...@gmail.com> >> wrote: >> >>> Thanks, we're trying that now! >>> >>> Russell Jurney @rjurney <http://twitter.com/rjurney> >>

Re: Simple Airflow BashOperators run but can't be scheduled or un-paused

Did you try to run this on Py 2.7 / 3.4 as well? I notice you are running on 3.6, which we are not testing against at the moment. Bolke. > On 16 May 2017, at 22:46, Russell Jurney wrote: > > We have tasks that run, but we can't get them to run as scheduled or to >

Re: Discussion on Airflow 1.8.1 RC2

7 at 10:55 AM, Alex Guziel <alex.guz...@airbnb.com > <mailto:alex.guz...@airbnb.com>> wrote: > I don't think any of the fixes I did were regressions. > > On Thu, May 4, 2017 at 8:11 AM, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdbr...@gmail.com>> wrote

Re: Discussion on Airflow 1.8.1 RC2

ot sure if their commits got > in before 1.8.1. > > On Wed, May 3, 2017 at 1:09 PM, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdbr...@gmail.com>> wrote: > Hi Dan, > > (Thread renamed to make sure it does not clash, dev@ now added) > > It surprises me th

Discussion on Airflow 1.8.1 RC2

solution for python > package isolation, and I'm working on integration with Kubernetes at the > executor level. > > Feel free to forward any of my messages to the dev mailing list. > > On Wed, May 3, 2017 at 11:18 AM, Bolke de Bruin <bdbr...@gmail.com > <mailto:bdb

Re: last task in the dag is not running

Hi Dmitry, Please provide more information, such as logs and the DAG definition itself. This is very little to go on unfortunately. Bolke > On 3 May 2017, at 10:22, Dmitry Smirnov wrote: > > Hi everyone, > > I'm using Airflow version 1.8.0, just upgraded from

Re: Force DAGs run up to the last task

Or use depends on past? Sent from my iPhone > On 28 Apr 2017, at 12:50, Jeremiah Lowin wrote: > > Hi David -- you'll want to set the concurrency parameter of your DAG to 1. > > J > >> On Fri, Apr 28, 2017 at 4:12 AM David Batista wrote: >> >> Hello

Re: dag file processing times

iguration. Our > longest-to-parse DAGs at Airbnb were reading yaml to build build a DAG, and > I believe someone wrote custom logic to avoid reparsing the yaml at every > cycle. Parsing equivalent json or hocon was an order of magnitude faster. > > Max > > On Mon, Apr 24, 20

Re: dag file processing times

at only exists in a >>> database and is retrieved by the Python code generating the DAG? >>> >>> Just asking because we have this case in production here. It is slowly >>> changing, so still fits within the Airflow framework, but you cannot just >>> wat

Re: dag file processing times

Inotify can work without a daemon. Just fire a call to the API when a file changes. Just a few lines in bash. If you bundle you dependencies in a zip you should be fine with the above. Or if we start using manifests that list the files that are needed in a dag... Sent from my iPhone > On 24

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

> On 23 Apr 2017, at 09:17, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > > Sent from my iPhone > >> On 23 Apr 2017, at 03:46, Hitesh Shah <hit...@apache.org> wrote: >> >> On Fri, Apr 21, 2017 at 8:19 AM, Chris Riccomini <criccom...@apac

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

Sent from my iPhone > On 23 Apr 2017, at 03:46, Hitesh Shah wrote: > > On Fri, Apr 21, 2017 at 8:19 AM, Chris Riccomini > wrote: > >> >>> Version in pkg-info has an rc0 notation. It should just be >> 1.8.1-incubating. >> >> This is a bit tricky to

Re: issue fetching master repo

Hi Boris, To be honest this is not an airflow question, but a git question. If you havent made any changes to the code, why don’t you delete the test folder and cone again? B. > On 20 Apr 2017, at 13:42, Boris Tyukin wrote: > > I just did this > > $ git clone

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

Hey Alex, I agree with you that they are nice to have, but as you mentioned they are not blockers. As we are moving towards time based releases I suggest marking them for 1.8.2 and cherry-picking them in your production. - Bolke. > On 18 Apr 2017, at 00:02, Alex Guziel

Re: 1.8.1 release update

Agree. Airflow-1000 can be merged. I think you want to put a big notice somewhere. Also some jiras still need to be cherry-picked into 1.8.1 and some will create conflicts. Bolke Sent from my iPhone > On 7 Apr 2017, at 20:16, Chris Riccomini wrote: > > Hey all, >

Re: PTAL: Airflow 2017 April Podling Report

Lgtm2 :-) Sent from my iPhone > On 5 Apr 2017, at 21:38, Chris Riccomini wrote: > > LGTM! Thanks! > > On Wed, Apr 5, 2017 at 11:57 AM, Gurer Kiratli < > gurer.kira...@airbnb.com.invalid> wrote: > >> Hi folks, >> >> Here is the draft of the podling report. Please take

Re: Podling Report Reminder - April 2017

:-) If you can please include our intention to graduate after the 1.8.1 release (no more reports ;-). B. > On 5 Apr 2017, at 19:32, Gurer Kiratli > wrote: > > I will do it. I will work on it today as it seems like today is the last > day to do this. > > On

Re: Google Summer of Code in Apache Airflow

Hey Jakub, Did you make any progress on this? Do you need any help/advice/assistance? - Bolke > On 8 Mar 2017, at 20:09, Jakub Powierza wrote: > > Hi Gerard, > Thanks for your reply! I was thinking about contribution connected with the > new DAGs UI or preparing

Re: 1.8.1 release

> I need people to review things. The more reviewers, the better. > > Also, can you please reach a conclusion on whether the issues you raised > (AIRFLOW-1019 and AIRFLOW-1013) are blockers or not (or just change the > status unilaterally if people aren't responding)? > > Chee

Re: 1.8.1 release

more, Sid, I checked AIRFLOW-1053 and while it is annoying I don’t think it is a blocker: it happens only with @once dags that have a SLA, hardly very common. Nevertheless a fix would be nice obviously. Bolke > On 3 Apr 2017, at 11:05, Bolke de Bruin <bdbr...@gmail.com> wrote: > &

Re: 1.8.1 release

che.org/jira/browse/AIRFLOW-1000 >>>> https://issues.apache.org/jira/browse/AIRFLOW-1001 >>>> https://issues.apache.org/jira/browse/AIRFLOW-1013 >>>> https://issues.apache.org/jira/browse/AIRFLOW-1018 >>>> https://issues.apache.org/jira/browse/AIR

Re: Scheduler silently dies

e scheduler with >>> "airflow >>>> scheduler", it's what gets printed on the console and starts >>>> with "Starting the scheduler". That is the file you want to >> investigate. >>>> >>>> If anything bad happens with

Re: Scheduler silently dies

e why tasks progress has become slow? But thats not >> the >>>> issue we are discussing here. So I am ignoring that here) >>>> >>>> 1.7: our prod is running 1.7 and we havent seen the "defunct process" >>>> issue for more than a week

Re: Scheduler silently dies

show memory limits, cpu count and airflow.cfg. Thanks Bolke > On 25 Mar 2017, at 18:16, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Please specify what “stop doing its job” means. It doesn’t log anything > anymore? If it does, the scheduler hasn’t died and hasn

Re: Scheduler silently dies

er that stopped > doing its job even after rebooting. I thought changing the start date or > the state of a task instance might be to blame but I've never been able to > pinpoint the problem either. > > We are using celery and docker if it helps. > > Le sam. 25 mars 2017 à 01:5

Re: Scheduler silently dies

7/collections.py", line 52, in __init__ > self.__update(*args, **kwds) File "/usr/lib/python2.7/_abcoll.py", line > 548, in update self[key] = value File "/usr/lib/python2.7/collections.py", > line 61, in __setitem__ last[1] = root[0] = self.__map[key] = [la

Re: 1.8.1 release

OW-1018. I > don't have log anymore :/ > > cheers, > > On Fri, Mar 24, 2017 at 6:59 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Hi Chris >> >> I think some jira are missing from the blocker list, I'll supply them >> soon. Also some fixes are alre

Re: Scheduler silently dies

ython2.7/_abcoll.py", line > 548, in update self[key] = value File "/usr/lib/python2.7/collections.py", > line 61, in __setitem__ last[1] = root[0] = self.__map[key] = [last, root, > key] File "/usr/local/lib/python2.7/dist-packages/airflow/utils/timeout.py", > line

Re: Scheduler silently dies

We are running *without* num runs for over a year (and never have). It is a very elusive issue which has not been reproducible. I like more info on this but it needs to be very elaborate even to the point of access to the system exposing the behavior. Bolke Sent from my iPhone > On 24 Mar

Re: 1.8.1 release

;>>>> https://issues.apache.org/jira/browse/AIRFLOW-983 >>>>> https://issues.apache.org/jira/browse/AIRFLOW-1019 (and in general >> the >>>>> slow >>>>> startup time from this new logic of orphaned/reset task) >>>>> https://issues.apache.org/jir

Re: 1.8.1 release

, Chris Riccomini <criccom...@apache.org> >> wrote: >> >>> Agreed. I'm looking for a list of checksums/JIRAs that we want in the >>> bugfix release. >>> >>> On Tue, Mar 21, 2017 at 12:54 PM, Bolke de Bruin <bdbr...@gmail.com> >>>

Re: 1.8.1 release

> On 21 Mar 2017, at 12:51, Bolke de Bruin <bdbr...@gmail.com> wrote: > > My suggestion, as we are using semantic versioning is: > > 1) no new features in the 1.8 branch > 2) only bug fixes in the 1.8 branch > 3) new features to land in 1.9 > > This al

Re: 1.8.1 release

My suggestion, as we are using semantic versioning is: 1) no new features in the 1.8 branch 2) only bug fixes in the 1.8 branch 3) new features to land in 1.9 This allows companies to fon Sent from my iPhone > On 21 Mar 2017, at 11:22, Chris Riccomini wrote: > > Hey

Re: [ANNOUNCE] Apache Airflow 1.8.0-incubating Released

1.8.0-incubating release and the > Airflow 1.8.0 release ? > > > > ________ > From: Bolke de Bruin <bdbr...@gmail.com> > Sent: Monday, March 20, 2017 5:30 PM > To: dev@airflow.incubator.apache.org; annou...@apache.org > Subject: [ANNOUNCE

[ANNOUNCE] Apache Airflow 1.8.0-incubating Released

The Apache Airflow (incubating) Team is proud to announce the release of Apache Airflow 1.8.0-incubating. This is a source code only release. ABOUT AIRFLOW Airflow is a platform to programmatically author, schedule and monitor workflows. Use airflow to author workflows as directed acyclic

Re: [RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

id >> wrote: > >> That's reasonable (treating it a bug instead of a change in behavior). Full >> speed ahead! >> >> On Thu, Mar 16, 2017 at 9:01 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: >> >>> Hello, >>> >>> Apache Airf

Re: SparkOperator - tips and feedback?

A spark operator exists as of 1.8.0 (which will be released tomorrow), you might want to take a look at that. I know that an update is coming to that operator that improves communication with Yarn. Bolke > On 18 Mar 2017, at 18:43, Russell Jurney wrote: > > Ruslan,

Re: `airflow webserver -D` runs in foreground

This is a (known) bug, since the introduction of the rolling restarts. Bolke. > On 17 Mar 2017, at 09:48, Ruslan Dautkhanov wrote: > > $ pip freeze > airflow==*1.8.0rc5*+apache.incubating > > airflow webserver doesn't want to daemonize > > > $ airflow webserver

Re: Airflow Committers: Landscape checks doing more harm than good?

We can do it in Travis’ afaik. We should replace it. So +1. B. > On 16 Mar 2017, at 16:48, Jeremiah Lowin wrote: > > This may be an unpopular opinion, but most Airflow PRs have a little red > "x" next to them not because they have failing unit tests, but because the >

Re: [VOTE] Release Apache Airflow 1.8.0 (incubating)

Oops, wrong mailing list ;-). > On 16 Mar 2017, at 09:28, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Hello Incubator PMC’ers, > > The Apache Airflow community has voted and approved the proposal to release > Apache Airflow 1.8.0 (incubating) based on 1.8.0 Rel

[VOTE] Release Apache Airflow 1.8.0 (incubating)

Hello Incubator PMC’ers, The Apache Airflow community has voted and approved the proposal to release Apache Airflow 1.8.0 (incubating) based on 1.8.0 Release Candidate 5. We now kindly request the Incubator PMC members to review and vote on this incubator release. If the vote is successful we

[RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

Hello, Apache Airflow (incubating) 1.8.0 (RC5) has been accepted. 9 “+1” votes received: - Maxime Beauchemin (binding) - Chris Riccomini (binding) - Arthur Wiedmer (binding) - Jeremiah Lowin (binding) - Siddharth Anand (binding) - Alex van Boxel (binding) - Bolke de Bruin (binding) - Daniel

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

gt; wrote: > > The only thing is that this is a change in semantics and changing semantics > (breaking some DAGs) and then changing them back (and breaking things > again) isn't great. > > On Wed, Mar 15, 2017 at 7:02 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >>

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

PR is available: https://github.com/apache/incubator-airflow/pull/2154 But marked for 1.8.1. - Bolke > On 15 Mar 2017, at 14:37, Bolke de Bruin <bdbr...@gmail.com> wrote: > > On second thought I do consider it a bug and can have a fix out pretty > quickly, but I don’t con

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

On second thought I do consider it a bug and can have a fix out pretty quickly, but I don’t consider it a blocker. - B. > On 15 Mar 2017, at 14:21, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Just to be clear: Also in 1.7.1 the DagRun was marked successful, but its &g

Re: Make Scheduler More Centralized

Hi Rui, We have been discussing this during the hackathon at Airbnb as well. Besides the reservations Gerard is documenting, I am also not enthusiastic about this design. Currently, the scheduler is our main issue in scaling. Scheduler runs will take longer and longer with more DAGs and more

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

>> >> On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >> >>> That is not an airflow error, but a Kerberos error. Try executing the >>> kinit command on the command line by yourself. >>> >>> Bolke

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

:). - Bolke > On 15 Mar 2017, at 09:04, Chris Riccomini <criccom...@apache.org> wrote: > > Has anyone been able to reproduce Sid's issue? > > On Tue, Mar 14, 2017 at 11:17 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> That is not an airflow error,

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

taging now. Fab work Bolke! >>>>>> -s >>>>>> >>>>>> On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov < >> dan.davy...@airbnb.com >>> . >>>>>> invalid >>>>>>> wrote: >>>>>> &

[VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

Dear All, Finally, I have been able to make the FIFTH RELEASE CANDIDATE of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys are available at

Re: Proposal to simplify start/end dates

lt_args start date to 2015 > - dagrun gets kicked off for 2015 > - User changes default_args start date to 2016 > - *dagruns start running for the 2016 start date instead of 2015* > >> On Tue, Mar 7, 2017 at 11:49 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: >>

Re: Proposal to simplify start/end dates

Hey Dan, Im not sure if I am seeing a difference for #1 vs now, except you are excluding backfills now from the calculation? Can you provide an example? Bolke > On 7 Mar 2017, at 20:38, Dan Davydov wrote: > > A very common source of confusion for our users is

Re: Help needed: Travis builds failing - psycopg2 and ldap3

and probably other variables suffer of the same issue. Cheers Bolke > On 6 Mar 2017, at 22:43, Bolke de Bruin <bdbr...@gmail.com> wrote: > > > Hint: It might be required to do a 'decode(“utf-8”)’ to make it pass. > > See the discussion on: https://github.com/cannatag/lda

Re: Help needed: Travis builds failing - psycopg2 and ldap3

Hint: It might be required to do a 'decode(“utf-8”)’ to make it pass. See the discussion on: https://github.com/cannatag/ldap3/issues/305 Bolke > On 6 Mar 2017, at 08:00, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Thanks! Much appreciated. > >> On 5 Mar 2017, at

Update status on getting to RC5 - reviewers wanted

Hi, Just wanted to do a short update on RC5 status. We had 8 (!) blockers, but the good news they are either fixed or a patch is available. The patches are in need of a review, so it would be appreciated if some of the committers can make some time available to do so. Please note that while

Re: Help needed: Travis builds failing - psycopg2 and ldap3

onnection function you mention in the ticket. > I think its really due to there difference between implementation of python > 2 vs python3 lib. > > anyway, I will look into it ( from this Tuesday) to see what kind of fix we > can put on Airflow side. > > Thanks > Jayesh > &

Re: Airflow running different with different user id ?

Nice management of expectations ;-). Sent from my iPhone > On 3 Mar 2017, at 21:44, Dan Davydov wrote: > > Within a couple of weeks. > >> On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong wrote: >> >> When approximately will it be released? >>

Re: Getting to RC5: Update

other > points you cc'd me on too. Thanks for continuing to drive this forward! > > On Wed, Mar 1, 2017 at 8:22 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Hi, >> >> Just wanted to give an update about the progress getting to RC5. As >> repor

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

, no as a workaround seems to exist. - Bolke > On 27 Feb 2017, at 23:19, Dan Davydov <dan.davy...@airbnb.com.INVALID> wrote: > > rc + your patch (and a couple of our own custom ones) > > On Mon, Feb 27, 2017 at 2:11 PM, Bolke de Bruin <bdbr...@gmail.com> wrote: > >> Da

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Dan Btw are you running with my patch for this? Or still plain rc? Cheers Bolke Sent from my iPhone > On 27 Feb 2017, at 22:46, Bolke de Bruin <bdbr...@gmail.com> wrote: > > I'll have a look. I verified and the code is there to take of this. > > B. > > Sent fro

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

s the subdag use case (setting tasks as successful via the graph view) > >> On Mon, Feb 27, 2017 at 11:06 AM, Bolke de Bruin <bdbr...@gmail.com> wrote: >> >> Hey Max >> >> It is massive for sure. Sorry about that ;-). However it is not as massive >> as you might d

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

ion] > it = IntegrationTestResults(dag_id='exmaple1') > assert it.ran_before('task1', 'task_2') > assert ti.overlapped('task1', 'task_3') # confirms 2 tasks ran in parallel > assert ti.none_failed() > assert ti.ran_last('root') > assert ti.max_concurrency_reached() == POOL_LIMIT

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

scenario (maybe when disk space issue?). I will add it back in. - Bolke > On 25 Feb 2017, at 09:07, Bolke de Bruin <bdbr...@gmail.com> wrote: > > Hi Dan, > > - Backfill indeed runs only one dagrun at the time, see line 1755 of jobs.py. > I’ll think about how to fix this o

Cutting down on testing time - updated

Hi All, (Welcome to new MacBook Pro that has a send “button” on the touch bar) Jeremiah and I have been looking into optimising the time that is spend on tests. The reason for this was that Travis’ runs are taking more and more time and we are being throttled by travis. As part of that we

Cutting down on testing time

Hi All, Jeremiah and I have been looking into optimising the time that is spend on tests. The reason for this was that Travis’ runs are taking more and more time and we are being throttled by travis. As part of that we enabled color coding of test outcomes and timing of tests. The results kind

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

example_short_circuit_operator', 'condition_is_True', datetime.datetime(2016, 1, 1, 0, 0))]) https://s3.amazonaws.com/archive.travis-ci.org/jobs/204780706/log.txt <https://s3.amazonaws.com/archive.travis-ci.org/jobs/204780706/log.txt> Bolke > On 25 Feb 2017, at 09:07, Bolke de Bruin <bdbr...@gmai

<    1   2   3   4   5   6   >