Re: Can I change the subject line for alert emails

I have a PR I need to rebase that would enable this (we had the exact same requirement): https://github.com/apache/incubator-airflow/pull/2338 Alek On Mon, Nov 12, 2018 at 12:40 PM muralisa...@gmail.com < muralisa...@gmail.com> wrote: > Hi, > > We have airflow running in different

Re: PSA: Make sure your Airflow instance isn't public and isn't Google indexed

This is a great idea, but we'd appreciate a setting that disables the banner even if those conditions aren't met - our instance is deployed without authentication, but is only accessible via our intranet. Alek On Tue, Jun 5, 2018, 3:35 PM James Meickle wrote: > I think that a banner

Re: Templatizing Email Content

I submitted a PR implementing this ( https://github.com/apache/incubator-airflow/pull/2338) a while ago, but it languished and I missed Fokko's request to revive it. I've rebased and it's ready for another review. Alek On Tue, May 29, 2018 at 10:08 AM vardangupta...@gmail.com <

Re: Airflow 2.0

We use sqlite for developing Airflow DAGs locally, and doing basic checks for syntax/import errors in our CI/CD pipeline. Alek On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini wrote: > Agree with Bolke that it's a good idea, but major work. I will push back on > this for

Re: DAG Factory Issues

n, > any module, and import is dependent on the python interpreter. > > If you know a way the suits every kind of structure please provide a pr. > > Cheers > Bolke > > Verstuurd vanaf mijn iPad > > > Op 5 dec. 2017 om 19:05 heeft Alek Storm <alek.st...@gmail.com> het

Re: DAG Factory Issues

We solved this (hackily) by setting the fileloc field on the generated DAG object to inspect.getsourcefile(inspect.stack()[1][0]), as the DAG constructor itself does. I agree a more general solution is needed; presumably the scheduler knows which Python file in the dags folder it was processing

Re: Data lineage and data portal

+1 On Wed, Nov 29, 2017 at 9:50 AM, Igors Vaitkus wrote: > +1 > > On 11/29/17, 3:48 PM, "Kate-Laurel Agnew" wrote: > > +1 > > On Wed, Nov 29, 2017 at 12:09 AM, Koen Mevissen > wrote: > > > +1 > > > >

Re: Ignore Processing DAG Definition Python Files for Paused DAGs

What's the advantage of this change? Performance? Alek On Mon, Nov 27, 2017 at 1:11 PM, ahu...@symphonyrm.com < ahu...@symphonyrm.com> wrote: > Hi all, > > I wanted to gauge community interest in this idea we have. We are > currently running a modified version of Airflow 1.9 RC3 where we ignore

Re: Display next schedule time and other schedule related information

I would greatly appreciate having some indication of the next run time of each DAG. In fact, some kind of Gantt chart showing all future DAG times together would be ideal. This would allow us to diagnose unknown resource contention between DAGs, which has bitten us before. Alek On Nov 24, 2017

Re: Airflow Deployment tools

Our TeamCity server detects the master branch has changed, then packages up the repo containing our DAGs as an artifact. We then use SaltStack to trigger a bash script on the targeted servers that downloads the artifact, moves the files to the right place, and restarts the scheduler (on the

Re: BashOperator exit status

> > [2017-11-14 15:09:38,277] {bash_operator.py:94} INFO - > > > pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112* > > > [2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Com

Re: BashOperator exit status

Hi Andy, The list doesn't allow inline images to be posted - can you paste your script content as text? Alek On Nov 14, 2017 9:16 AM, "Andrew Maguire" wrote: > Hi, > > I have some bash operators that are failing but airflow is not picking the > failure up. > > Here is

Re: [VOTE] Airflow 1.9.0rc1

; > Bolke. > > > > > > > On 9 Nov 2017, at 15:53, Arthur Wiedmer <art...@apache.org> wrote: > > > > > > > > Hi Alek, > > > > > > > > Technically, we cannot release a distribution on PyPI until we have > > vot

Re: [VOTE] Airflow 1.9.0rc1

I think this has been mentioned before, but it would be much easier for us (my team) to test RCs if they were published to PyPI. Or is that against Apache guidelines? Alek On Thu, Nov 9, 2017 at 8:29 AM, Michael Crawford < michael.crawf...@modernizingmedicine.com> wrote: > Thanks. Yes I

Re: Exception in worker when pickling DAGs

, 2017 at 2:04 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > FYI: there's been talks on deprecating pickling altogether as it's very > brittle. > > Max > > On Thu, Oct 12, 2017 at 10:45 AM, Alek Storm <alek.st...@gmail.com> wrote: > > > Can anyone

Re: Exception in worker when pickling DAGs

Can anyone help with this? Has anyone successfully used Airflow with pickling turned on that can give details on their setup? Thanks, Alek On Mon, Oct 9, 2017 at 2:00 PM, Alek Storm <alek.st...@gmail.com> wrote: > Yes, everything's correctly imported - everything works fine w

Exception in worker when pickling DAGs

Hi all, When running the scheduler as airflow scheduler -p and the worker (on a different box) as airflow worker -q foo, I get the following exception in the worker process when triggering a DAG run manually: Traceback (most recent call last): File "/site/var/airflow/venv/bin/airflow", line

Re: Run job multiple times with same execution date

01-01 11:00 > 2017-01-02 14:00 2017-01-02 11:00 2017-01-02 14:00 2017-01-01 14:00 > 2017-01-02 17:00 2017-01-02 14:00 2017-01-02 17:00 2017-01-01 17:00 > > On Thu, Apr 27, 2017 at 4:04 PM, Alek Storm <alek.st...@gmail.com> wrote: > > > It's `0 2-20/3 * * *`. The p

Run job multiple times with same execution date

I'd like to create a job that runs six times a day, all with the same execution date (the previous day). Is there a way to do this without generating six separate DAGs, or running an `airflow clear` command in between runs? Thanks, Alek

Re: Schedule tasks on following weekday

Can anyone help with this? I'd prefer to avoid performing complicated date logic in my operators to derive the correct execution date. Thanks, Alek On Sun, Mar 19, 2017 at 11:54 AM, Alek Storm <alek.st...@gmail.com> wrote: > Hi all, > > I need to schedule daily jobs to be run o