Re: [VOTE] Airflow 1.10.0rc1

[Bolke de Bruin]

Kaxil,

Please vote with binding / non-binding. This is important for the IPMC vote 
that won’t know all committers.

B.

Verstuurd vanaf mijn iPad

> Op 10 jul. 2018 om 03:22 heeft Naik Kaxil  het volgende 
> geschreven:
> 
> +1
> 
> Regards,
> Kaxil
> 
> On 10/07/2018, 02:09, "Kengo Seki"  wrote:
> 
>+1 (non-binding)
> 
>- Checked signatures and checksums
>- Confirmed the source distribution corresponds to v1-10-test's HEAD
>(f1083cb).
>- Installed the binary distribution with cryptography package ran some
>example dags with no error
>- Checked version info via WebUI and confirmed RBAC worked
> 
>Kengo Seki apache.org>
> 
>2018-07-09 5:02 GMT+09:00 Bolke de Bruin :
> 
>> Hey all,
>> 
>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
>> which will last for 72 hours. Consider this my (binding) +1.
>> 
>> Airflow 1.10.0 RC 1 is available at:
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>> 
>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
>> comes with INSTALL instructions.
>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>> "sdist"
>> release.
>> 
>> Public keys are available at:
>> 
>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>> 
>> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>> 
>> Please note that the version number excludes the `rcX` string as well
>> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
>> to rename the artifact without modifying the artifact checksums when we
>> actually release.
>> 
>> 
>> Cheers,
>> Bolke
> 
> 
> 
> 
> 
> 
> Kaxil Naik 
> 
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK 
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com

Scheduler not honouring non_pooled_task_slot_count config

[ramandumcs]

We are using airflow version 1.9 with celery executor. And we are observing 
that Airflow Scheduler is not honouring the "non_pooled_task_slot_count" 
config.  We are using default setting which is set to 128. But we could 
schedule and run >128 tasks concurrently.
>From code it seems that scheduler is re-initialising the open_slots with 128 
>instead of setting the remaining left over slots.
In jobs.py
for pool, task_instances in pool_to_task_instances.items():
if not pool:
# Arbitrary:
# If queued outside of a pool, trigger no more than
# non_pooled_task_slot_count per run
open_slots = conf.getint('core', 'non_pooled_task_slot_count')
Thanks,
Raman Gupta

Re: [VOTE] Airflow 1.10.0rc1

[Naik Kaxil]

+1

Regards,
Kaxil

On 10/07/2018, 02:09, "Kengo Seki"  wrote:

+1 (non-binding)

- Checked signatures and checksums
- Confirmed the source distribution corresponds to v1-10-test's HEAD
(f1083cb).
- Installed the binary distribution with cryptography package ran some
example dags with no error
- Checked version info via WebUI and confirmed RBAC worked

Kengo Seki apache.org>

2018-07-09 5:02 GMT+09:00 Bolke de Bruin :

> Hey all,
>
> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the 
release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 1 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>
> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the 
changelog.
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
>
> Cheers,
> Bolke






Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: [VOTE] Airflow 1.10.0rc1

[Kengo Seki]

+1 (non-binding)

- Checked signatures and checksums
- Confirmed the source distribution corresponds to v1-10-test's HEAD
(f1083cb).
- Installed the binary distribution with cryptography package ran some
example dags with no error
- Checked version info via WebUI and confirmed RBAC worked

Kengo Seki apache.org>

2018-07-09 5:02 GMT+09:00 Bolke de Bruin :

> Hey all,
>
> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 1 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>
> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
> "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the changelog.
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
>
> Cheers,
> Bolke

Re: Apache Airflow 1.10.0b3

[Taylor Edmiston]

We discussed the splitting out of hooks & operators from core internally at
Astronomer again today.  I should be able to allocate some time to
splitting out hooks & operators for the purpose of speeding up the CI.  A
faster CI on core would be hugely beneficial for all contributors.

I created a Jira issue for this at https://issues.apache.org/
jira/browse/AIRFLOW-2732.

If anyone has pointers for this work, I'm all ears.  I've done a lot with
Python unit tests in the past but this part of the Airflow codebase is new
to me.

Taylor

*Taylor Edmiston*
Blog  | CV
 | LinkedIn
 | AngelList
 | Stack Overflow



On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin  wrote:

> Separating the tests is where the effort lies. So having that as a
> consequence of splitting the packages would be nice. It has come up a
> couple of times but it was not picked up unfortunately.
>
> B.
>
> > On 28 Jun 2018, at 08:32, Maxime Beauchemin 
> wrote:
> >
> > It would be so nice to have a fast test suite. Having to wait for Travis
> > for up to an hour makes many workflows (like working on a release) super
> > painful.
> >
> > I spoke with folks at Astronomer recently about moving all operators and
> > hooks to another Python package that airflow would import. This would
> allow
> > for independent test suites and to have a more regular release cadence on
> > hooks and operators. What do you think?
> >
> > Max
> >
> > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin 
> wrote:
> >
> >> Arghhh. The downside of doing this late at night and wanting to go to
> >> bed... :-). Will make a new one
> >>
> >> Sent from my iPhone
> >>
> >>> On 28 Jun 2018, at 00:07, Chris Fei  wrote:
> >>>
> >>> Great, thank you! I just took this for a quick spin and it looks like
> >>> there's DB migration task missing. The task you committed just
> recently,
> >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
> >>> which can't be found when running airflow initdb (seehttps://
> >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
> migrations/versions
> >> ).
> >>> Chris
> >>>
> >>>
>  On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
>  Hi All,
> 
>  I have created a sdist package that is available at:
> 
> 
> >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> bating.tar.gz
>  <
> >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> bating.tar.gz>>
> >>
>  In order to distinguish it from an actual (apache) release it is:
> 
>  1. Marked as beta (python package managers do not install beta
>   versions by default - PEP 440)> 2. It is not signed
>  3. It is not at an official apache distribution location
> 
>  You can also put something like this in a requirements.txt file:
> 
>  git+
> 
> >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3
>  <
> >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3
> >>>
>  ]>   airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
> s,slack,s-
>  3]
>  <
> >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3][1]
> >>>
> >>
>  and then "pip install -r requirements.txt”.
> 
>  I hope that after this beta we can go to RC and start voting on 1.10.>
>  Cheers
>  Bolke
> >>>
> >>>
> >>> Links:
> >>>
> >>> 1.
> >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> airflow@master#egg=apache-airflow[celery,crypto,emr,
> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> 
> >>  che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> airflow@master%23egg=apache-airflow[celery,crypto,emr,
> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> >>
>
>

PR for refactoring Airflow SLAs

[James Meickle]

Hi folks,

Based on my earlier email to the list, I have submitted a PR that splits
`sla=` into three independent SLA parameters, as well as heavily
restructuring other parts of the SLA feature:

https://github.com/apache/incubator-airflow/pull/3584

This is my first Airflow PR and I'm still learning the codebase, so there's
likely to be flaws with it. But I'm most interested in the general
compatibility of this feature with the rest of Airflow. We want this for
our purposes at Quantopian, but we'd really prefer to get it into Airflow
core rather than running a fork forever!

Let me know your thoughts,

-James M.

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Ash Berlin-Taylor]

Great! Thanks for doing this. I've left some review comments on your PR.

-ash

> On 9 Jul 2018, at 11:45, Verdan Mahmood  wrote:
> 
> ​Hey Guys, ​
> 
> In an effort to simplify the JS dependencies of Airflow
> ​​
> ,
> ​I've
> introduce
> ​d​
> npm and webpack for the package management. For now, it only implements
> this in the www_rbac version of the web server.
> ​
> 
> Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> 
> The problem with the
> ​existing ​
> frontend (
> ​JS
> ) code of Airflow is that most of the custom JS is written
> ​with​
> in the html files, using the Flask's (Jinja) variables in that JS. The next
> step of this effort would be to extract that custom
> ​JS
> code in separate JS files
> ​,​
> use the dependencies in those files using require or import
> ​ and introduce the JS automated test suite eventually. ​
> (At the moment, I'm simply using the CopyWebPackPlugin to copy the required
> dependencies for use)
> ​.
> 
> There are also some dependencies which are directly modified in the codebase
> ​ or are outdated​
> . I couldn't found the
> ​ correct​
> npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> Apparently dagre-d3.js that we are using is one of the gist or is very old
> version
> ​ not supported with webpack 4​
> , while the gantt-chart-d3v2 has been modified according to Airflow's
> requirements
> ​ I believe​
> .
> ​ Used the existing libraries for now. ​
> 
> ​I am currently working in a separate branch to upgrade the DagreD3
> library, and updating the custom JS related to DagreD3 accordingly. ​
> 
> This PR also introduces the pypi_push.sh
> 
> script that will generate all the JS statics before creating and uploading
> the package.
> ​
> ​Please let me know if you guys have any questions or suggestions and I'd
> be happy to answer that. ​
> 
> Best,
> *Verdan Mahmood*
> (+31) 655 576 560

Airflow's JS code (and dependencies) manageable via npm and webpack

[Verdan Mahmood]

​Hey Guys, ​

In an effort to simplify the JS dependencies of Airflow
​​
,
​I've
 introduce
​d​
 npm and webpack for the package management. For now, it only implements
this in the www_rbac version of the web server.
​

Pull Request: https://github.com/apache/incubator-airflow/pull/3572

The problem with the
​existing ​
frontend (
​JS
) code of Airflow is that most of the custom JS is written
​with​
in the html files, using the Flask's (Jinja) variables in that JS. The next
step of this effort would be to extract that custom
​JS
 code in separate JS files
​,​
use the dependencies in those files using require or import
​ and introduce the JS automated test suite eventually. ​
(At the moment, I'm simply using the CopyWebPackPlugin to copy the required
dependencies for use)
​.

There are also some dependencies which are directly modified in the codebase
​ or are outdated​
. I couldn't found the
​ correct​
npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
Apparently dagre-d3.js that we are using is one of the gist or is very old
version
​ not supported with webpack 4​
, while the gantt-chart-d3v2 has been modified according to Airflow's
requirements
​ I believe​
.
​ Used the existing libraries for now. ​

​I am currently working in a separate branch to upgrade the DagreD3
library, and updating the custom JS related to DagreD3 accordingly. ​

This PR also introduces the pypi_push.sh

script that will generate all the JS statics before creating and uploading
the package.
​
​Please let me know if you guys have any questions or suggestions and I'd
be happy to answer that. ​

Best,
*Verdan Mahmood*
(+31) 655 576 560

Re: Timezone support on Web UI

[Bolke de Bruin]

Of course just didn’t get to it yet. PRs are also welcome btw!

Verstuurd vanaf mijn iPad

> Op 9 jul. 2018 om 08:05 heeft Manu Zhang  het 
> volgende geschreven:
> 
> Hi all,
> 
> We've been using Airflow for job scheduling for quite a while. It's been
> great except that everything is in UTC time (We are from China and in
> GMT+8) . We are pretty excited when timezone support is added on latest
> master and try upgrading. Nonetheless, as in the doc
> , "Please
> note that the Web UI currently only runs in UTC", the blocking issue is
> that Web UI has no timezone support yet. It will be very confusing for
> users when they've defined their DAGs with local time while the DAGs'
> metrics are displayed in UTC time.
> 
> Is there a plan to add timezone support to Web UI ?
> 
> Thanks,
> Manu Zhang

Timezone support on Web UI

[Manu Zhang]

Hi all,

We've been using Airflow for job scheduling for quite a while. It's been
great except that everything is in UTC time (We are from China and in
GMT+8) . We are pretty excited when timezone support is added on latest
master and try upgrading. Nonetheless, as in the doc
, "Please
note that the Web UI currently only runs in UTC", the blocking issue is
that Web UI has no timezone support yet. It will be very confusing for
users when they've defined their DAGs with local time while the DAGs'
metrics are displayed in UTC time.

Is there a plan to add timezone support to Web UI ?

Thanks,
Manu Zhang