Re: s3 logging issues

Does your bucket name has dots(.) ? On Fri, Jun 10, 2016 at 7:14 AM, Jeremiah Lowin wrote: > Jason, > > I will try to figure this out for you tomorrow. > > Jeremiah > > On Thu, Jun 9, 2016 at 12:20 PM Jason Kromm > wrote: > > > Hi all, > > > >

Re: Upgrade from 1.7.0 to 1.7.1.2

afaik you have to run "airflow upgradedb" command, so any new migrations will be applied. On Fri, Jun 10, 2016 at 12:35 PM, Jason Chen wrote: > Hi, > > I upgraded airflow from 1.7.0 to 1.7.1.2 (using "pip install airflow > --upgrade"). > > Do I need to any "post

Re: ImportError: cannot import name MySqlOperator

mysql_hook uses MySQLdb. Just see if you are not hit by this issue: http://stackoverflow.com/questions/6383310/python-mysqldb-library-not-loaded-libmysqlclient-18-dylib On Mon, Jun 20, 2016 at 5:58 AM, Msr Msr wrote: > Hi, > > I have installed Airflow on Mac and trying

Long pending PR

Hello guys, There has been a PR long pending for review, can some one please take a look on it and merge it? https://github.com/apache/incubator-airflow/pull/1682/files There is a very small change in core files and all changes are in contrib section only, which too are mainly for fixing pep8

Re: NL Airflow meetup in Amsterdam July 19th

Hi, Is it possible to present remotely, as if there are any open slots? Thanks, Sumit On Tue, Jul 12, 2016 at 1:33 AM, Bolke de Bruin wrote: > Hi All, > > Just wanted to let you know that GoDataDriven will be hosting us in their > Amsterdam offices for an Airflow meetup

Re: svn commit: r1756572 - /incubator/public/trunk/content/projects/airflow.xml

original) > +++ incubator/public/trunk/content/projects/airflow.xml [utf-8] Wed Aug 17 > 03:25:44 2016 > @@ -1,4 +1,4 @@ > - > +?xml version="1.0" encoding="UTF-8"?> >

Re: About airflow scheduler

Hi Jason, See this from the conf file: # The scheduler can run multiple threads in parallel to schedule dags. # This defines how many threads will run. However airflow will never # use more threads than the amount of cpu cores available. max_threads = 2 So probably the no of processes you see

Re: Call to Committers : PR Clean up Duty : ETA Nov 15

Hi Sid, I've also picked up my share. Thanks, Sumit On Fri, Oct 7, 2016 at 8:11 AM, siddharth anand wrote: > Folks! > > I've changed the title of the wiki to "*Cold Case PR Resolution*" > https://cwiki.apache.org/confluence/display/AIRFLOW/ > Cold-Case+PR+Resolution. > >

Re: Airflow Meetup tomorrow

Big +1 for live streaming.. On Tue, Sep 20, 2016 at 11:14 PM, siddharth anand wrote: > Jeff, > For the meet-up tomorrow, I'm assuming there will be a streaming feed and > possibly also a video conferencing component since a lot of the committers > and community members are

Re: Regarding force running the whole graph from UI

Twinkle, I think while clearing a task u can press upstream and downstream buttons, this will clear all tasks of that dag run and will eventually re-run whole dag. On Fri, Oct 21, 2016 at 4:29 PM, twinkle sachdeva < twinkle.sachd...@gmail.com> wrote: > @Shin: Nice find. > > @Alex : Thanks for

Re: Simple Feature Request

+1.. nice to have it.. On Thu, Nov 24, 2016 at 3:51 AM, siddharth anand wrote: > If we support a test query dialog, we could just execute the query via each > hook's pre-existing execute method. > > -s > > On Wed, Nov 23, 2016 at 2:04 PM, Bolke de Bruin

Cron expression with depends_on_past causing no TIs to be generated

Found that on our current prod version (1.7.0) and master as well, scheduler doesn't fire TIs if *scheduler_interval* is a cron expression and *depends_on_past* is True. The very simple DAG I used for testing can be found here: http://pastebin.com/wPnFjRwD However it creates new DAG runs as per

Re: Cron expression with depends_on_past causing no TIs to be generated

o track this issue. > > > > Bolke. > > > > > >> On 14 Jan 2017, at 12:43, Sumit Maheshwari <sumeet.ma...@gmail.com> > wrote: > >> > >> Found that on our current prod version (1.7.0) and master as well, > >> scheduler doesn't fir

Re: Reminder : LatestOnlyOperator

Vincent, I think "dag.catchup = False" affects the whole DAG, means skipping all tasks in it. While "LatestOnlyOperator" can be used to skip only some of the tasks in a DAG as well. On Wed, Mar 22, 2017 at 7:05 PM, Vincent Poulain < vincent.poul...@tinyclues.com> wrote: > I did not see the

Re: Welcome @saguziel as a committer and PMC member!

Welcome Alex.. On Mon, Apr 17, 2017 at 10:33 PM, siddharth anand wrote: > Welcome Alex! > > On Fri, Apr 14, 2017 at 8:23 AM, Chris Riccomini > wrote: > > > Congrats, Alex! Welcome. :) > > > > On Thu, Apr 13, 2017 at 7:06 PM, Dan Davydov

Re: [VOTE] Release Airflow 1.8.1 RC1

+1 (binding) On Mon, Apr 24, 2017 at 10:09 PM, Chris Riccomini wrote: > Dear All, > > I've made Airflow 1.8.1 RC1 available at: > https://dist.apache.org/repos/dist/dev/incubator/airflow, public keys are > available at

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC4

+1 (binding) Thanks, Sumit Maheshwari cell. 9632202950 On Thu, Aug 17, 2017 at 3:49 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Ok done, the whole process is very prone to error, hopefully I did > everything right. > > Max > > On Wed, Aug 16, 2017 at 2:

Re: Airflow + Kubernetes discussion

I would join as well for sure. Thanks, Sumit Maheshwari cell. 9632202950 On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <criccom...@apache.org> wrote: > I would definitely be up to joining. We're interested in the K8s work > that's going on. That time works for me. > > On

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

; Hello, > > > > Apache Airflow (incubating) 1.8.2 (RC2) has been accepted. > > > > 5 “+1” votes received, 4 out of which are "binding": > > > > - Maxime Beauchemin (binding) > > - Bolke de Bruin (binding) > > - Chris Riccomini (bind

Re: Info needed regarding upgrading to 1.8.2

or changing > definitions of new dags). When I want to see dag updated on UI, I hit > refresh button but this is just for UI to show it properly. No restarts > needed in our experience with 1.8.1 > > On Fri, Aug 18, 2017 at 6:40 AM, Sumit Maheshwari <sumeet.ma...@gmail.com>

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

+1, binding. On Mon, Jun 26, 2017 at 3:49 PM, Bolke de Bruin wrote: > We have been running it for the last couple of days. No issues and seems > more responsive. > > +1, binding > > Bolke > > > On 25 Jun 2017, at 01:10, Maxime Beauchemin > wrote:

Re: Active Branch for development

It should be master only. On Thu, Sep 14, 2017 at 8:16 PM, Muhammad Ahmmad wrote: > Hello, > > I'm working on a fix for AIRFLOW-1013 apache.org/jira/browse/AIRFLOW-1013> and AIRFLOW-1055 apache.org/jira/browse/AIRFLOW-1055>. > > And I

Re: Airflow Webserver behind a reverse proxy

Sergei, Check the first bullet/point on this blog <https://www.qubole.com/blog/creating-apache-airflow-as-a-service/>, it might help you. Thanks, Sumit Maheshwari cell. 9632202950 On Thu, Sep 14, 2017 at 2:34 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > It wo

Re: Airflow 1.8.2 released

Awesome!! Thanks a lot Max for being the RM for this release. On Mon, Sep 4, 2017 at 10:51 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Dear Airflow community, > > Airflow 1.8.2 was just released. > > The source release as well as the binary "sdist" release are available >

Intermittent failures of docker operator test cases

Hi All, Recenlty Im seeing intermittent faiures of DockerOperatorTestCase in Travis, though my code have not even touching that area. The error I am seeing in all below failed cases is this ImportError : cannot import name Client And these are the testcases which are failing: test_execute

Re: PR Review Request

two remarks on the code. I still have to fire up Airflow to see > how this would work in the UI. > > Cheers, Fokko > > 2017-11-06 8:07 GMT+01:00 Sumit Maheshwari <sumeet.ma...@gmail.com>: > > > Hi All, > > > > As of now TI model view in Airflow is very st

Re: PR Review Request

I hope that some more folks have it reviewed. So can we merge this now, and start using this feature. We can of course create followup jiras/PRs if needed. Thoughts? On Sat, Dec 9, 2017 at 2:10 AM, Russell Pierce wrote: > It looks like

PR Review Request

Hi All, As of now TI model view in Airflow is very static and each operator has to make use of given options only. I have opened a PR to add support for more links (buttons) on model view, which can redirect users to the outside of Airflow programmatically. Some simple use cases of this feature

Re: PR Review Request

e he is not able to convince me that UI processes 1 task node at a time, so there would be no extra load on the web server. On Thu, Nov 23, 2017 at 7:12 PM, Sumit Maheshwari <sumeet.ma...@gmail.com> wrote: > Ping! > > folks, please review :) > > > On Mon, Nov 6, 2017 at 12:

Xcom_push is broken - probably after the timezone awareness

Folks, I've some testes, which used to pass earlier, but recently they started failing. When I looked closely it appeared that xcom_push is behaving differently. So this is is the current code, which throws the error mentioned below to it DEFAULT_DATE = datetime(2017, 1, 1) ti =

Re: Xcom_push is broken - probably after the timezone awareness

tch that sets the default timezone for TaskInstance is also helpful in > this case and will be welcomed. How to do it you can find in the DAG class. > > Bolke. > > On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.ma...@gmail.com> wrote: > > Folks, > > I've some test

Re: ImportError: No module named sendgrid

er >> > https://github.com/apache/incubator-airflow/commit/ >> > 7cb818bbacb2a2695282471591a9e323d8efbf5c <https://github.com/apache/ >> > incubator-airflow/commit/7cb818bbacb2a2695282471591a9e323d8efbf5c> >> > > 3, 4, 5. I agree, and I made the same point

ImportError: No module named sendgrid

<https://issues.apache.org/jira/browse/AIRFLOW-1723>. Now couple of doubts: 1. Thanks, Sumit Maheshwari cell. 9632202950

Re: ImportError: No module named sendgrid

. On Wed, Oct 25, 2017 at 2:12 PM, Sumit Maheshwari <sumeet.ma...@gmail.com> wrote: > When I fetched the latest master, installed it and ran webserver, it > failed with this error: > > *ImportError: No module named sendgrid* > > On further investigation, I found that it has

Re: Facing SSL issue with dev/airflow-pr tool

it, > > I think it is unrelated to the PR tool. Please check the below link: > > https://stackoverflow.com/questions/44316292/ssl- > sslerror-tlsv1-alert-protocol-version > > Try running: pip install requests[security] > > Regards, > Kaxil > > On 15/05/2018,

Re: Q1 Airflow Bay Area Meetup

> > We can present our internal tool for doing data quality checks for ETL's > in Qubole. > Iris is an Airflow Operator that currently has built-in functions for count > validation, data-profile validation and duplicate data validation. That would be great Devjyoti. folks, I would really love

Re: PR Review Request

Happy new year guys.. plz review this PR <https://github.com/apache/incubator-airflow/pull/2657> and give your +1s or -1s, so we can either merge it or discard it.. Thanks in advance!! On Thu, Dec 14, 2017 at 4:56 PM, Sumit Maheshwari <sumeet.ma...@gmail.com> wrote: > I hope

Re: Airflow Meetup in India

Hi Raman, Folks from Qubole certainly join/talk if the meetup is happening in Bangalore. However, sometime back I did a poll about the same and got only 7 responses, so have dropped the idea of conducting. Let us know if you get enough traction this time. Thanks, Sumit On Tue, Aug 21, 2018 at

Re: Airflow Meetup in India

t; Thanks, > Raman Gupta > > On 2018/08/21 06:38:37, Sumit Maheshwari wrote: > > Hi Raman, > > > > Folks from Qubole certainly join/talk if the meetup is happening in > > Bangalore. However, sometime back I did a poll about the same and got > only > > 7 re

Re: Airflow Meetup in India

Yes, the poll was conducted around March end. On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra wrote: > Forms says preferred month is may, june or july. Is the form outdated? > > On Tue 21 Aug, 2018, 17:43 Sumit Maheshwari, > wrote: > > > Sure, here it is https://goo.gl/fo

Re: Airflow Meetup in India

a speaker for airflow meetup > > On Tue, 21 Aug 2018 at 19:45, Sumit Maheshwari > wrote: > > > @Vaibhav, I guess if there are atleast 20 or plus people are interested, > > then only it makes sense to conduct a meetup. > > > > @Sai, have updated the form. > > >

Re: Airflow Meetup in India

ving few introductary sessions so as to reach bigger audience. Clearly > > there may not be many indians/ residents of bangalore in this mail list. > > Shall we few series of airflow meetups. > > > > On Sun 26 Aug, 2018, 22:34 Sumit Maheshwari, > > wrote: > >

Re: Python 3.6 Support for Airflow 1.10.0

Strong +1 for keeping 2.7 for now. We've 30+ big and small enterprises running Airflow as a service on our platform and they all are on py 2.7 for now. Soon we are adding support for py 3.5 as well, but still, not everyone gonna switch to that immediately. So IMO we should keep the support of py

Re: Airflow Meetup in India

ttps://cwiki.apache.org/confluence/display/AIRFLOW/Announcements >- https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups >- this DL and our twitter channel > > -s > > On Sun, Aug 26, 2018 at 10:22 PM Sumit Maheshwari > wrote: > > > Thanks Sid. > >

Re: Airflow Meetup in India

ere we can plan a airflow meetup. > > On Tue 21 Aug, 2018, 18:01 Sumit Maheshwari, > wrote: > > > Yes, the poll was conducted around March end. > > > > > > > > On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra > > wrote: > > > > >

Re: [Live Webinar Tomorrow] AIRFlow at Scale - Register Now

d be very > interested in a recording or some other option that doesn't require waking > up at 5:30 AM my time, as the content sounds great! :) > > On Tue, Jul 10, 2018 at 7:58 AM, Sumit Maheshwari > wrote: > > > FYI: > > > > > > -- Forwarded message

Re: 1.10.0beta1 now available for download

Great work Fokko and Bolke, really really appreciated!! On Mon, Apr 23, 2018 at 3:38 PM, Guillermo Rodríguez Cano wrote: > Awesome! Thanks for the efforts! Time to test lots of things! > > > On Mon, Apr 23, 2018 at 9:45 PM, Chris Riccomini > wrote: >

A new blog on Airflow

An awesome blog is written by one of my colleague at Qubole. Hope it'll help people in setting up Airflow for their prod ETL use cases. https://www.qubole.com/blog/hood-building-air-qubole/ Thanks, Sumit

Airflow meetup in Bangalore (India)

Hi folks, Please fill out the small google form (attached below), if you are interested in attending/presenting in Airflow meetup. If there is a good amount of interest (15-20 pax at least), then I'll take care of arranging the meetup. https://goo.gl/forms/2NF6HtUwpOOTvi373 Thanks, Sumit

Re: Airflow meetup in Bangalore (India)

meet-ups : > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups > -s > > On Tue, Mar 27, 2018 at 3:01 AM, Sumit Maheshwari <msu...@apache.org> > wrote: > > > Hi folks, > > > > Please fill out the small google form (attached below), if you

Re: RBAC Update

Awesome.. Great work indeed.. On Sat, Mar 24, 2018, 13:43 Bolke de Bruin wrote: > Hear hear! > > Bolke. > > > On 24 Mar 2018, at 06:01, Arthur Wiedmer > wrote: > > > > Thanks a lot for all the effort Joy! > > > > Great job! > > > > Best, > > Arthur

Re: Airflow Meetup in India

Hi Sai, Yes, we are planning for the same, though details haven't been finalized yet. On Sat, Sep 22, 2018 at 11:46 PM Sai Phanindhra wrote: > Hi sumit, can we have airflow meetup in october? > > On Mon 27 Aug, 2018, 12:10 Sumit Maheshwari, > wrote: > > > Sure,

Re: Tail log instead of serving all logs for long running tasks/huge log volume

Tried to solve that problem (partially) with below PR https://github.com/apache/incubator-airflow/pull/1875 On Tue, Sep 25, 2018 at 3:18 PM Sai Phanindhra wrote: > Hi guys, >Now we serve whole log file irrespective of its size. Sometimes when i > log size if few Mb(more than some 5mb or

Re: Slides from London Airflow Meetup #1

Thanks a lot, Ash. On Tue, Sep 25, 2018 at 3:47 PM Ash Berlin-Taylor wrote: > Hi everyone, > > Slides from the first London Airflow Meetup are available in this this > Google drive folder: > https://drive.google.com/drive/folders/1wiSkrg_1rvqGrmbYN7rNFQaqW0Ty10Vk < >

Re: Tail log instead of serving all logs for long running tasks/huge log volume

w N lines but lets implement it on > worker end to fetch last N lines. > > On Tue 25 Sep, 2018, 15:37 Sumit Maheshwari, > wrote: > > > Tried to solve that problem (partially) with below PR > > > > https://github.com/apache/incubator-airflow/pull/1875 > > &g

Re: Graduation resolution passed - Airflow is a TLP

Hurrah.. Congrats to whole community.. On Fri, Dec 21, 2018, 08:32 Deng Xiaodong Congratulations! > > XD > > On Fri, Dec 21, 2018 at 06:18 Tao Feng wrote: > > > Thanks Jakob for driving the graduation! Great news! > > > > On Thu, Dec 20, 2018 at 1:13 PM Jakob Homan wrote: > > > > > Hey all- >