Re: Airflow Webserver behind a reverse proxy

Hi Sergei, You can also remove gunicorn completly and let Nginx talk to the Flask app directly using wsgi: https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-14-04 I'm not super familiar with the routing in Nginx. A subdomain like

Re: 1.9.0 test branch has been cut

Hi all, I'm working on the Airflow logging. Got a few PR's in there, but there still a few open ends that I would like to have into 1.9: AIRFLOW-1604 is mostly cosmetic and almost finished. I think it is important to get AIRFLOW-1611

Re: Airflow stops reading stdout of forked process with BashOperator

Hi David, Thank you for the question. The problem for the Spark-sql hook seems related , but the issue is different. At the spark-sql hook, the problem was that there where two iterators, one for the stdout, and one for the stderr. First the one

Re: Return results optionally from spark_sql_hook

n 600 dags like that every > morning. > > I am thinking now to create a pyspark job that will do insert and count and > write it to a temp file. Still not ideal... I wish I could just parse spark > SQL instead.. > > On Oct 14, 2017 8:05 AM, "Driesprong, Fokko"

Re: Return results optionally from spark_sql_hook

> On Oct 15, 2017 05:04, "Driesprong, Fokko" <fo...@driesprong.frl> wrote: > > > Hi Boris, > > > > Instead of writing it to a file, you can also write it to xcom, this will > > keep everything inside of Airflow. My personal opinion on this; spark-sql >

Re: new logging

Hi Boris, Can you please elaborate? Where did the output used to end up? Did you provide any custom logging config, or are you using the default config? Maybe drop some example code. Cheers, Fokko 2017-10-16 19:02 GMT+02:00 Boris : > Hi guys, > > I used to do

Re: new logging

t; > 'owner': 'airflow', > > 'start_date': airflow.utils.dates.days_ago(2) > > } > > > > dag = DAG( > > dag_id='python_test_logging', default_args=args, > > schedule_interval=None) > > > > def print_context(ds, **kwargs): > > pprint(kwar

Re: Airflow 1.9 Logging Set Up

Hi Andy, Thanks for reaching out. We are debugging the new logging, and input from the community is highly appreciated. If you are using Python 2, you'll need to put an empty __init__.py in all the directory, so ~/airflow/plugins/__init__.py, this needs to be empty. Could you share your

Re: Return results optionally from spark_sql_hook

Hi Boris, Thank you for your question and excuse me for the late response, currently I'm on holiday. The solution that you suggest, would not be my preferred choice. Extracting results from a log using a regex is expensive in terms of computational costs, and error prone. My question is, what

Re: spark sql hook with multiple queries

Hi Boris, Interesting. Multiple queries is supported by the spark-sql operator and this should work using Airflow. Executing SQL from a file: Fokkos-MBP:~ fokkodriesprong$ spark-sql --driver-java-options "-Dlog4j.configuration=file:///tmp/log4j.properties" -f query.sql 1 Time taken: 1.976

Re: Return results optionally from spark_sql_hook

rows and then do ALTER TABLE EXCHANGE > PARTITION in the same pyspark job but I found out that spark does not > support this statement yet and I have to use Hive. > > On Sat, Oct 14, 2017 at 4:53 AM, Driesprong, Fokko <fo...@driesprong.frl> > wrote: > > > Hi Boris, >

Re: Airflow in different server

Hi Jagdish, You can do this in two ways: - Using the SSHHook, you can ssh to another machine, and execute shell scripts there. - This requires installing Airflow-scheduler on the other machines. But you can install the Celery executor to scale out Airflow:

Re: Issues with postgres_operator

Hi Shoumitra, So if I understand correctly, the query finishes but the connection does not close. What is the result of the query? Maybe you can also share some code of how you use the PostgresOperator the communicate with Redshift. Regarding your config. The sequential executor is only

Re: Airflow 1.9.0 status

Hi All, I would like to include AIRFLOW-1611 in the 1.9.0 release: https://github.com/apache/incubator-airflow/pull/2631 Currently importing a custom logging configuration is not work (as far as I know). Any feedback on the PR would also be appreciated. Cheers, Fokko 2017-09-25 23:27

Re: Apache Airflow welcome new committer/PMC member : Fokko Driespong (a.k.a. fokko)

Thanks for the welcome, great to be part of this! Cheers, Fokko 2017-10-06 10:51 GMT+02:00 Sumit Maheshwari : > Welcome Fokko :) > > > > On Thu, Oct 5, 2017 at 5:54 AM, Boris Tyukin > wrote: > > > great news! > > > > On Wed, Oct 4, 2017 at 6:03

Re: Airflow 1.9 Logging Set Up

ot;, > line 47, in configure_logging > 'Unable to load custom logging from {}'.format(logging_class_path) > ImportError: Unable to load custom logging from > plugins.airflow_logging_settings.LOGGING_CONFIG > > So don't think my error even got that far. > > Happy to help

Re: Airflow 1.9 Logging Set Up

Hi all, I'll push a fix today. I've created a Jira ticket: https://issues.apache.org/jira/browse/AIRFLOW-1731 Cheers, Fokko 2017-10-17 19:27 GMT+02:00 Driesprong, Fokko <fo...@driesprong.frl>: > Hi Andy, > > I see something weird. While debugging the code, the path

Re: Enabling flake8

Hi Bolke, Great initiative. This should also help improve the quality of the PR's. Let me know if you need any help. Cheers, Fokko 2017-11-27 16:47 GMT+01:00 Bolke de Bruin : > Hi All, > > I just wanted to let you know, I am working on enabling flake8 tests. It > should only

Re: [VOTE] Airflow 1.9.0rc4

+1 Op ma 27 nov. 2017 om 21:24 schreef Bolke de Bruin > +1, binding > > Let’s get it out! > > Sent from my iPhone > > > On 27 Nov 2017, at 20:32, Chris Riccomini wrote: > > > > Hey all, > > > > I have cut Airflow 1.9.0 RC4. This email is calling a vote

Re: [VOTE] Airflow 1.9.0rc8

+1 binding Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin > +1, binding > > Checked sigs, version, source is there (did not check build), bin is there. > > Bolke > > Verstuurd vanaf mijn iPad > > > Op 15 dec. 2017 om 23:31 heeft Joy Gao het volgende >

Re: [VOTE] Airflow 1.9.0rc6

+1 from my side Cheers, Fokko Op di 12 dec. 2017 om 17:28 schreef Ash Berlin-Taylor < ash_airflowl...@firemirror.com> > +0.5 from me. > > Our big test will come on Thursday morning, but looking good so far for > the small daily dags we've got are running okay, logs are showing up, and > making

Re: [VOTE] Airflow 1.9.0rc3

+1, Binding Cheers, Fokko 2017-11-18 20:01 GMT+01:00 Bolke de Bruin : > +1, Binding > > > > On 18 Nov 2017, at 18:36, Chris Riccomini wrote: > > > > Hey all, > > > > I have cut Airflow 1.9.0 RC3. This email is calling a vote on the > release, > > which

Re: RBAC Update

Good morning, The demo provided by Max is down, it throws a SQLAlchemyexception :'( Cheers, Fokko 2017-11-18 19:14 GMT+01:00 Chris Riccomini : > @bolke, open issues on the Github repo, please. > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin >

Re: Airflow Deployment tools

I'm using Ansible to deploy the Airflow, the steps are: - First install Airflow using pip (or a rc using curl) - Do an `airflow version` to trigger the creation of the default config - Set the config correctly variables in the config using Ansible. - Deploy the supervisord files - Start everything

Re: [VOTE] Airflow 1.9.0rc2

I have RC2 running for a couple of days without experiencing any issues. +1, binding Cheers, Fokko 2017-11-15 14:48 GMT+01:00 Bolke de Bruin : > Hi Chris, > > Your changelog seems incomplete (some more fixes were included in RC2). > > But apart from that > > +1, binding > >

Dutch Airflow meetup

Hi all, Since Airflow is really picking up in The Netherlands, we've started an Airflow meetup. Please sign up if you are interested in attending an Airflow related meetup: https://www.meetup.com/Amsterdam-Airflow-meetup/ A first edition will be announced on short notice. Kind regards, Fokko

Re: PR Review Request

Hi Sumit, Thanks for the PR. I think this is a nice addition. This would also be applicable for the Google Cloud and Databricks operators. I've had two remarks on the code. I still have to fire up Airflow to see how this would work in the UI. Cheers, Fokko 2017-11-06 8:07 GMT+01:00 Sumit

Docs

Hi All, I got a message from a colleague regarding the docs. The docs page is quite old: https://airflow.apache.org/installation.html Which suggests to install airflow instead of apache-airflow. When looking into the Github, everything looks fine:

Re: [VOTE] Airflow 1.9.0rc1

Hi Michael, You have to install it from the tar.gz: wget https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc1/apache-airflow-1.9.0rc1+incubating-bin.tar.gz pip install /tmp/apache-airflow.tar.gz The steps of updating, are in the UPDATING.md:

Re: [VOTE] Airflow 1.9.0rc1

Hi all, +1 (binding) I got Airflow 1.9rc1 deployed today, had some delay. So far it works great, except one thing. I've got some weird stuff in my log, but I'm not sure if it is Airflow, or a misbehaving process on my side: [2017-11-08 16:31:05,842] {logging_mixin.py:91} WARNING - Traceback

Re: Airflow talk @ Coolblue

Hi Gerard, Thanks for sharing the presentation. Unfortunately I could not make it to the make it to the presentation. Will there be a follow-up? For example thinks that your team encountered when migrating from Azkaban to Airflow? Kind regards, Fokko Driesprong 2017-10-29 11:29 GMT+01:00 Gerard

Re: Airflow Docker Container

Hi Daniel, My dear colleague from GoDataDriven, Bas Harenslak, started on building an official Docker container on the Dockerhub. I've put him in the CC. In the end I strongly believe the image should end up in the official Docker repository: https://github.com/docker-library/official-images

Re: 答复: Airflow REST API proof of concept.

Hi Luke, This is the REST api for the new UI: https://github.com/apache/incubator-airflow/blob/master/airflow/www_rbac/api/experimental/endpoints.py RBAC = Role Based Access Control, the fine grained security model based on the fabmanager. Recently we've added some endpoints to it. In the end

Re: Airflow with Celery

I had similar issues with Airflow running the Celery executor. The celery_result_backend should be a persistent database like Postgres or MySql. What broker are you using? I would recommend using Redis or RabbitMQ, depending on what you like the most. Cheers, Fokko 2018-05-15 21:12 GMT+02:00

Re: Python3 and sensors module

Hi Cindy, The other sensors should work under Python3. We try to support Python3 as much as possible, but sometimes libraries are used that are not compatible. Could you describe what you are running into? Cheers, Fokko 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis :

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

>> 3CCAO2yDybRq2VUM1JYo_6VT_H8Ca7Lu8af6H-2CZKQzYT6xYGU-g%40mail.gmail.com > %3E > >>> > >>> > >>>> On 6 Mar 2018, at 09:57, Ash Berlin-Taylor > <ash_airflowlist@firemirror. > >>> com> wrote: > >>>> > >>&g

Re: 1.10.0 Release

Hi Kaxil, Good point. Right now Bolke and I are a bit busy with the PyData talk that we're giving tomorrow in Amsterdam. I've did a `git checkout v1-10-test && git reset --hard apache/master` a couple of times to make it up to date with master. We have to be careful to set the version number

Re: Master is broken

Hi Gerardo, I totally agree that when master turns red, we should stop merging and fix the build or revert the commit that broke the build. I think one of the underlying problems is having flaky tests, I tried to fix a few of those, but they are quite persistent. Sometimes it is hard to

Re: Airflow 1.10.0

> > > > Sorry that I've been MIA for a while. Just wanted to check in on 1.10, > and > > its current status. > > > > Cheers, > > Chris > > > > On Tue, May 1, 2018 at 11:18 PM Driesprong, Fokko > > wrote: > > > >> Hi Bryon, > &

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

Edmiston* > Blog <https://blog.tedmiston.com/> | CV > <https://stackoverflow.com/cv/taylor> | LinkedIn > <https://www.linkedin.com/in/tedmiston/> | AngelList > <https://angel.co/taylor> | Stack Overflow > <https://stackoverflow.com/users/149

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

Hi Craig, This is something that needs to be fixed. I agree with you this is very dirty. In your installation you're not installing the kubernetes stuff, so the KubernetesPodOperator is ignored. We need to figure out how to have example dags that are not compatible with the vanilla installation,

Re: How to wait for external process

Hi Stefan, Afaik there isn't a more efficient way of doing this. DAGs that are relying on a lot of sensors are experiencing the same issues. The only way right now, I can think of, is doing updating the state directly in the database. But then you need to know what you are doing. I can image that

Re: HttpSensor raising exception with status=403

Hi Pedro, You could just create a CustomHttpHook and place it on your pythonpath, then you should also create a CustomHttpSensor. Hope this helps. Cheers, Fokko 2018-05-26 2:48 GMT+02:00 Pedro Machado : > Hi, > > I am using HttpSensor to look for a file. The webserver is

Re: Airflow 1.10.0

Hi Bryon, We'll be releasing the RC's soon. If there aren't much issues with the RC's, it will be released quickly. But we need the community to test these. Cheers, Fokko 2018-05-01 20:57 GMT+02:00 Wicklund, Bryon : > Hey I was wondering if you had a date in mind or

Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes

Also one of the main benefits of the Kubernetes Executor is having a Docker image that contains all the dependencies that you need for your job. Personally I would switch to Kubernetes when it leaves the experimental stage. Cheers, Fokko 2018-04-28 16:27 GMT+02:00 Kyle Hamlin

Re: Problem with SparkSubmit

Hi Anton, Which version of Airflow are you running? Cheers, Fokko 2018-04-27 10:24 GMT+02:00 Anton Mushin : > Hi all, > I have problem with spark operator. I get exception > > user@host:/# airflow test myDAG myTask 2018-04-26 > [2018-04-26 15:32:11,279] {driver.py:120}

Re: Airflow 1.9.0 status

gt;>> stdout > > >>>>>>>> AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() > > >>> exception > > >>>>>>> for @once > > >>>>>>>> AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeate

Re: Airflow 1.9 Logging Set Up

> just figure out airflow.cfg and get things up and running. > > Maybe the plan is to cut down some of these steps once the new logging is > more final but just wanted to give a little bit of feedback - being able to > just set everything up via airflow.cfg is great. > > Cheers, &

Re: Fix on_kill command for operators

Yes, for Spark this should work. Depending on the operator and the implementation: https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428 However this is a big change in behaviour. I'm curious about the

Re: Airflow 1.10

I think 1.10 is a good idea. I'm working on this refactoring of the sensor structure: https://github.com/apache/incubator-airflow/pull/2875 Would be awesome to get this in. At my current project we use sensors in a few places, but still there is some work to be done. For example, don't allocate

Re: Airflow Documentation - Readthedocs

This is a good question, I see that artwr and Maxime are owners of the project, maybe they can fix the build? Or add some more Airflow committers as readthedocs project owners. I get quite some questions from colleagues because they are reading old docs :) Cheers, Fokko 2018-02-04 14:32

Fixed Travis permission problem

Hi all, I've hotfixed the CI permission problem: Exception: Traceback (most recent call last): File "/home/travis/build/apache/incubator-airflow/.tox/py35-backend_mysql/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File

Re: Scheduling error that seems to be related to @once since Airflow v1.9 upgrade

Hi Aaron, It looks like you hit a bug: https://issues.apache.org/jira/browse/AIRFLOW-1977 Currently I don't have a solution for this at hand, I have to dig into the code. If the DAG is scheduled one, you might want to try setting it to None instead of @once:

Re: Airflow Documentation - Readthedocs

gt; > > >>>>> Best, > > >>>>> Arthur > > >>>>> > > >>>>> On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin < > > >> bdbr...@gmail.com> > > >>>> wro

Re: Airflow Documentation - Readthedocs

e GitHub home (Readme) are all broken. > > Should I change the all the links to readthedocs? Or can someone who > has access to PyPI create the docs? > > What do you suggest? > > Regards, > Kaxil > > On 11/02/2018, 06:03, "Dri

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

Welcome Ash! 2018-02-23 18:45 GMT+01:00 Andy Hadjigeorgiou : > Congrats Ash! > > > On Feb 23, 2018, at 12:19 PM, Taylor Edmiston > wrote: > > > > Congrats, Ash! Very exciting. > > > > *Taylor Edmiston* > > TEdmiston.com |

Re: Airflow 1.9.0 is released

Awesome work! Cheers, Fokko Op wo 3 jan. 2018 om 18:50 schreef Chris Riccomini > Hey all, > > I have updated the docs as well: > > https://airflow.incubator.apache.org/ > > Cheers, > Chris > > On Wed, Jan 3, 2018 at 9:30 AM, Sid Anand wrote: > > >

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

Welcome Feng! Awesome to have you on board! 2018-08-03 10:41 GMT+02:00 Naik Kaxil : > Hi Airflow'ers, > > > > Please join the Apache Airflow PMC in welcoming its newest member and > > co-committer, Feng Tao (a.k.a. feng-tao). > > > > Welcome Feng, great to have you

Re: [VOTE] Airflow 1.10.0rc3

+1 Binding Installed it using: SLUGIFY_USES_TEXT_UNIDECODE=yes pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz Cheers, Fokko 2018-08-03 9:47 GMT+02:00 Bolke de Bruin : > Hey all, > > I have cut Airflow 1.10.0 RC3.

Re: Kerberos and Airflow

Hi Ry, You should ask Bolke de Bruin. He's really experienced with Kerberos and he did also the implementation for Airflow. Beside that he worked also on implementing Kerberos in Ambari. Just want to let you know. Cheers, Fokko Op do 26 jul. 2018 om 23:03 schreef Ry Walker > Hi everyone - > >

Re: We've migrated to Github to repo!

Hi Max, You're right. I just started plowing though my mailbox and merged a commit without squash and merge, but it changes history as you mention. Nice thing of Github is if you change it, it remembers your preference which is Squash and Merge :-) Love the Gitbox so far, great work! Cheers,

Re: We've migrated to Github to repo!

low debate > here...] > > Max > > On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko > wrote: > > > Hi Max, > > > > You're right. I just started plowing though my mailbox and merged a > commit > > without squash and merge, but it changes history as you

Re: [VOTE] Airflow 1.10.0rc4

-1 (binding) Sorry Bolke for not checking this earlier. In rc3 we've replaced some of the reserved keywords. But still I'm unable to run a simple dag in the 1.10 rc4 release under Python 3.7: MacBook-Pro-van-Fokko:sdh-api-pobt fokkodriesprong$ docker run -e SLUGIFY_USES_TEXT_UNIDECODE=yes -t -i

Re: [VOTE] Airflow 1.10.0rc4

Good point Bolke, Sid, seems that there are still a few issues with Tenacity as well , therefore I would like to change my vote: +1 (binding) Cheers, Fokko 2018-08-09 14:08 GMT+02:00 Ash Berlin-Taylor : > +0.5 (binding) from me. > > Tested

Re: PR Review Dashboard?

Hi Holden, Thanks for reaching out. Recently we've moved to Apache Gitbox ( https://gitbox.apache.org/), so we use the Github UI directly instead of having to merge using a CLI ( https://github.com/apache/incubator-airflow/blob/master/dev/airflow-pr). Not sure if we're already up to the game of

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

t; wrote: > > > > > > > > > > > Congratulations, Feng! > > > > > > > > > > > > *Taylor Edmiston* > > > > > > Blog <https://blog.tedmiston.com/> | CV > > > > > > <https://stack

Re: [RESULT][VOTE] Release Airflow 1.10.0

Thanks Bolke! I've just pushed the artifacts to Apache Dist: https://dist.apache.org/repos/dist/release/incubator/airflow/1.10.0-incubating/ I don't have any access to pypi, this means that I'm not able to upload the artifacts over there. Anyone in the position to grand me access or upload it to

Re: Jira cleanup and triage

Hi Gerardo, Thanks for bringing this up. This is actually a good point. Recently we've moved the Apache Airflow repo to the Gitbox repo ( https://gitbox.apache.org/). Before with the Apache repo, the Github repo was just a mirror of the Apache one. Now we do everything on Github itself. We still

Re: PR Review Dashboard?

s week > and I’ll report back. > > On Tue, Aug 7, 2018 at 12:15 AM Driesprong, Fokko > wrote: > > > Hi Holden, > > > > Thanks for reaching out. Recently we've moved to Apache Gitbox ( > > https://gitbox.apache.org/), so we use the Github UI directly instead o

Re: [RESULT][VOTE] Release Airflow 1.10.0

Thanks Max, My PyPI ID is Fokko Cheers, Fokko 2018-08-21 22:49 GMT+02:00 Maxime Beauchemin : > I can, what's your PyPI ID? > > Max > > On Mon, Aug 20, 2018 at 2:11 PM Driesprong, Fokko > wrote: > > > Thanks Bolke! > > > > I've just pushed the

Amsterdam Apache Airflow meetup

Hi all, Due a cancellation there is a speaker slot available at the Amsterdam Meetup the 12th of September: https://www.meetup.com/Amsterdam-Airflow-meetup/events/253673642/ If you're interested in speaking let me know. It can be quite broad, from how you're using Airflow within your

Re: [RESULT][VOTE] Release Airflow 1.10.0

gt; B. > > Sent from my iPhone > > > > > Kaxil Naik > > Data Reply > 2nd Floor, Nova South > 160 Victoria Street, Westminster > London SW1E 5LB - UK > phone: +44 (0)20 7730 6000 > k.n...@reply.com > www.reply.com > On 22 Aug 2018, at 16:13, Driespr

Re: WebHdfsSensor doesn't support HDFS HA

Hi Manu, Thanks for raising this question. There is a PR for moving to hdfs3. There is code in the existing codebase, which support HA

Use Docker for running Airflow tests

Hi all, Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR is still open here: https://github.com/apache/incubator-airflow/pull/3393 To make this work, and not store any Docker images in private repositories, I've created a new repository which stores the Docker image

Re: [RESULT][VOTE] Release Airflow 1.10.0

Certainly: https://github.com/apache/incubator-airflow/releases/tag/1.10.0 Cheers, Fokko Op wo 22 aug. 2018 om 15:18 schreef Ash Berlin-Taylor : > Could you push the git tag too please Fokko/Bolke? > > -ash > > > On 22 Aug 2018, at 08:16, Driesprong, Fokko > wrot

Re: Regarding airflow 1.10

Hi Hemanth, Thanks for the question. There is the KubernetesExecutor which just uses a container with Airflow to schedule a task on the kubernetes cluster. This container will then execute a single task, and then the Executor checks the exitcode of the container. There is also the

Re: updating dags best practice

Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy the master branch. This is with a LocalExecutor. Cheers, Fokko Op di 21 aug. 2018 om 15:25 schreef Dev

Re: [RESULT][VOTE] Release Airflow 1.10.0

wrote: > > Done for Bolke, Fokko and kaxil. > > Best, > Arthur > > On Sun, Aug 26, 2018 at 3:08 AM Driesprong, Fokko > > wrote: > > > Gentle ping! Would be awesome to get 1.10 on Pypi :-) > > > > Op wo 22 aug.

Re: [Proposal] Explicit re-schedule of sensors

Thanks Stefan for picking this up. The sensors are in desperate need for some redesign for the aforementioned reasons. Please note this ticket: https://issues.apache.org/jira/browse/AIRFLOW-2001 It addresses the same issue. Regarding the open question. I would be reluctant for introducing new

Re: [VOTE] Airflow 1.10.0rc1

does not work yet, but it isn't supported anyway. I would like to have https://github.com/apache/incubator-airflow/pull/3604 in RC2. It isn't a critical bug, but it looks messy. Cheers, Fokko 2018-07-14 0:06 GMT+02:00 Driesprong, Fokko : > Thanks Bolke for all the effort. > > I t

Re: [VOTE] Airflow 1.10.0rc2

My vote is ​+1 (binding)​ Cheers, Fokko 2018-07-16 9:52 GMT+02:00 Driesprong, Fokko : > Awesome Bolke! > > I don't have a production Airflow at hand right now, but I've ran some > simple tests against Python 2.7, 3.5, 3.6 and it all looked fine. > > +1 From my side > > C

Re: [VOTE] Airflow 1.10.0rc2

Awesome Bolke! I don't have a production Airflow at hand right now, but I've ran some simple tests against Python 2.7, 3.5, 3.6 and it all looked fine. +1 From my side Cheers, Fokko 2018-07-15 20:05 GMT+02:00 Bolke de Bruin : > Hey all, > > I have cut Airflow 1.10.0 RC2. This email is calling

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

​Nice work Verdan. The frontend really needed some love, thank you for picking this up. Maybe we should also think deprecating the old www. Keeping both of the UI's is something that takes a lot of time. Maybe after the release of 1.10 we can think of moving to Airflow 2.0, and removing the old

Re: Call for fixes for Airflow 1.10.1

Can we add this one as well? https://github.com/apache/incubator-airflow/pull/3862 https://issues.apache.org/jira/browse/AIRFLOW-1917 I'm happy to cherry pick them onto the 1.10.1 by myself as well. Any idea when we will start this branch? Cheers, Fokko Op do 6 sep. 2018 om 08:08 schreef Deng

Re: re run build

Hi, This PR has to rebase onto master first to get rid of the conflicts. Cheers, Fokko Op wo 5 sep. 2018 om 09:20 schreef airflowuser : > Can someone assist with rerun build for: > https://github.com/apache/incubator-airflow/pull/2488 > > tried to send it to committers mailing list but for

Re: DagRunOperator - target dag tasks are not triggered - possible timezone conflict

Hi Niranda, What version of Airflow are you running? There are a lot of improvements coming up in Airflow 1.10 regarding timezones. https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L77-L79 Cheers, Fokko 2018-07-05 14:57 GMT+02:00 Niranda

Re: Apache Airflow 1.10.0b3

Hi all, I just tried to install a local Airflow instance within Docker, using Python 3.5, but I got an error on initdb. Anyone else experiencing this? It has to do with the schema migrations. Cheers, Fokko 2018-07-09 22:56 GMT+02:00 Taylor Edmiston : > We discussed the splitting out of hooks &

Re: Apache Airflow 1.10.0b3

Ah, I picked the wrong tar.gz, Thanks B Cheers, Fokko 2018-07-13 15:07 GMT+02:00 Bolke de Bruin : > Please use the Rc Fokko. This has been fixed in the RC. > > B. > > Verstuurd vanaf mijn iPad > > > Op 13 jul. 2018 om 15:01 heeft Driesprong, Fokko > het volgende

Re: [VOTE] Airflow 1.10.0rc1

I've installed Airflow, but using a clean install and the sequential executor, the tasks are not being picked up. This is easy to replicate: docker run -t -i --rm -p 8080:8080 python:3.5 bash pip install

Re: [VOTE] Airflow 1.10.0rc1

With the SequentialExecutor the webserver also acts as the scheduler (without parallelism) 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson : > Need to run the scheduler also right? > > -- > Carl Johan Gustavsson > > On 13 July 2018 at 15:37:22, Driesprong, Fokko (fo...@dries

Re: [VOTE] Airflow 1.10.0rc1

re not picked up. If you put a dag in the normal dag > folder it works fine. > >> > >> Please create a jira for this @fokko. A pr would be appreciated. > >> > >> B. > >> > >> Sent from my iPhone > >> > >>> On 13 Jul 2

Re: [VOTE] Airflow 1.10.0rc1

> Verstuurd vanaf mijn iPad > > > Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko > het volgende geschreven: > > > > Ok, I've did some testing. > > > > 1.10 works fine with the LocalExecutor. With the SequentialExecutor it > does > > not pick up any task, e

Re: Airflow 1.8 CeleryExecutor: multiple servers in broker_url

Hi Andrii, When looking in the Airflow code, I'd say this should workd: https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/executors/celery_executor.py#L41 Which version of celery are you using? Please note, when you are updating to 1.9.1, the config of Celery changes to make

Re: Call for fixes for Airflow 1.10.1

nsure that he/she knows what the state is. > > > > B > > > > Op zo 9 sep. 2018 18:02 schreef Driesprong, Fokko >: > > Can we add this one as well? > > > > https://github.com/apache/incubator-airflow/pull/3862 < > https://github.com/apache/inc

Re: Very looong py files

Maybe wait until this one has been merged 浪 https://github.com/apache/incubator-airflow/pull/3116/files Cheers, Fokko Op zo 11 mrt. 2018 om 21:34 schreef Driesprong, Fokko <fo...@driesprong.frl> > Hi Bruno, > > I agree that there are files that are too big as you menti

Re: Very looong py files

Hi Bruno, I agree that there are files that are too big as you mentioned. Sometimes structures are being refactored like I did with the sensors. The problem is that if you refactor one of these files, most PR requests will have merge conflicts. But this should not be seen as an impediment to

Re: Readthedocs

Thanks Kaxil! Cheers, Fokko 2018-04-10 14:19 GMT+02:00 Naik Kaxil <k.n...@reply.com>: > Added you. > > https://readthedocs.org/projects/airflow/ > > Regards, > Kaxil > > > On 10/04/2018, 13:04, "fo...@driesprongen.nl on behalf of Driesprong, > Fokk

Re: Give it up for Fokko!

Thanks you all for the kind words. I really love the energetic community around Airflow and all the cool stuff that we're working on. I find it truly amazing how we build such an awesome product with great people from all around the world! Cheers, Fokko 2018-04-14 1:56 GMT+02:00 Sid Anand

Readthedocs

Hi all, Would it be possible to give me access to the Readthedocs environment. I'm working on improving the docs a bit, but I can't see the output of the docs generation. Cheers, Fokko

Re: Issue : Merges are not syncing back to master

Thanks for picking this up Sid! Cheers, Fokko 2018-04-06 7:22 GMT+02:00 Sid Anand : > Heads up. Apache Airflow github master > is behind Apache Airflow > master >

  1   2   >