Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Bolke de Bruin]

O that’s great stuff. Hopefully it works nicely with Keycloak. Did you try by 
any chance? Why not upstream this?

> On 7 Aug 2018, at 11:55, Ravi Kotecha  wrote:
> 
> Slightly OT but we've done some work on auth with FAB that might be useful
> for anyone using OpenIDConnect and www_rbac here:
> https://github.com/ministryofjustice/fab-oidc
> 
> It could serve as a template if people are doing something different too
> 
> On Mon, Jul 23, 2018 at 6:50 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
> 
>> Are there any gaps on the new vs old UI at this point?
>> 
>> For many that upgrade will require some work to set up authentication on
>> the new UI. This is fairly well documented in FAB.
>> 
>> Max
>> 
>> On Mon, Jul 23, 2018 at 3:08 AM Bolke de Bruin  wrote:
>> 
>>> I think it should be removed now. 1.10.X should be the last release seri
>> s
>>> that supports the old www. Do we need to vote on this?
>>> 
>>> Great work Verdan!
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
 Op 23 jul. 2018 om 10:23 heeft Driesprong, Fokko >> 
>>> het volgende geschreven:
 
 ​Nice work Verdan.
 
 The frontend really needed some love, thank you for picking this up.
>>> Maybe
 we should also think deprecating the old www. Keeping both of the UI's
>> is
 something that takes a lot of time. Maybe after the release of 1.10 we
>>> can
 think of moving to Airflow 2.0, and removing the old UI.
 
 
 Cheers, Fokko​
 
 2018-07-23 10:02 GMT+02:00 Naik Kaxil :
 
> Awesome. Thanks @Verdan
> 
> On 23/07/2018, 07:58, "Verdan Mahmood" 
>>> wrote:
> 
>   Heads-up!! This frontend change has been merged in master branch
> recently.
>   This will impact the users working on Airflow RBAC UI only. That
>>> means:
> 
>   *If you are a contributor/developer of Apache Airflow:*
>   You'll need to install and build the frontend packages if you want
>> to
> run
>   the web UI.
>   Please make sure to read the new section, "Setting up the node /
>> npm
>   javascript environment"
>    CONTRIBUTING.md#setting-up-the-node--npm-javascript-
> environment-only-for-www_rbac>
> 
>   in CONTRIBUTING.md
> 
>   *If you are using Apache Airflow in your production environment:*
>   Nothing will impact you, as every new build of Apache Airflow will
> come up
>   with pre-built dependencies.
> 
>   Please let me know if you have any questions. Thank you
> 
>   Best,
>   *Verdan Mahmood*
> 
> 
>   On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
>   maximebeauche...@gmail.com> wrote:
> 
>> Glad to see this is happening!
>> 
>> Max
>> 
>> On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
>> ash_airflowl...@firemirror.com> wrote:
>> 
>>> Great! Thanks for doing this. I've left some review comments on
> your PR.
>>> 
>>> -ash
>>> 
 On 9 Jul 2018, at 11:45, Verdan Mahmood <
> verdan.mahm...@gmail.com>
>>> wrote:
 
 ​Hey Guys, ​
 
 In an effort to simplify the JS dependencies of Airflow
 ​​
 ,
 ​I've
 introduce
 ​d​
 npm and webpack for the package management. For now, it only
> implements
 this in the www_rbac version of the web server.
 ​
 
 Pull Request: https://github.com/apache/
> incubator-airflow/pull/3572
 
 The problem with the
 ​existing ​
 frontend (
 ​JS
 ) code of Airflow is that most of the custom JS is written
 ​with​
 in the html files, using the Flask's (Jinja) variables in that
> JS. The
>>> next
 step of this effort would be to extract that custom
 ​JS
 code in separate JS files
 ​,​
 use the dependencies in those files using require or import
 ​ and introduce the JS automated test suite eventually. ​
 (At the moment, I'm simply using the CopyWebPackPlugin to copy
> the
>>> required
 dependencies for use)
 ​.
 
 There are also some dependencies which are directly modified in
> the
>>> codebase
 ​ or are outdated​
 . I couldn't found the
 ​ correct​
 npm versions of those libraries. (dagre-d3.js and
> gantt-chart-d3v2.js).
 Apparently dagre-d3.js that we are using is one of the gist or
> is very
>>> old
 version
 ​ not supported with webpack 4​
 , while the gantt-chart-d3v2 has been modified according to
> Airflow's
 requirements
 ​ I believe​
 .
 ​ Used the existing libraries for now. ​
 
 ​I am currently working in a separate branch to upgrade the
> DagreD3
 library, and updating the custom JS related 

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Ravi Kotecha]

Slightly OT but we've done some work on auth with FAB that might be useful
for anyone using OpenIDConnect and www_rbac here:
https://github.com/ministryofjustice/fab-oidc

It could serve as a template if people are doing something different too

On Mon, Jul 23, 2018 at 6:50 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Are there any gaps on the new vs old UI at this point?
>
> For many that upgrade will require some work to set up authentication on
> the new UI. This is fairly well documented in FAB.
>
> Max
>
> On Mon, Jul 23, 2018 at 3:08 AM Bolke de Bruin  wrote:
>
> > I think it should be removed now. 1.10.X should be the last release seri
> s
> > that supports the old www. Do we need to vote on this?
> >
> > Great work Verdan!
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 23 jul. 2018 om 10:23 heeft Driesprong, Fokko  >
> > het volgende geschreven:
> > >
> > > ​Nice work Verdan.
> > >
> > > The frontend really needed some love, thank you for picking this up.
> > Maybe
> > > we should also think deprecating the old www. Keeping both of the UI's
> is
> > > something that takes a lot of time. Maybe after the release of 1.10 we
> > can
> > > think of moving to Airflow 2.0, and removing the old UI.
> > >
> > >
> > > Cheers, Fokko​
> > >
> > > 2018-07-23 10:02 GMT+02:00 Naik Kaxil :
> > >
> > >> Awesome. Thanks @Verdan
> > >>
> > >> On 23/07/2018, 07:58, "Verdan Mahmood" 
> > wrote:
> > >>
> > >>Heads-up!! This frontend change has been merged in master branch
> > >> recently.
> > >>This will impact the users working on Airflow RBAC UI only. That
> > means:
> > >>
> > >>*If you are a contributor/developer of Apache Airflow:*
> > >>You'll need to install and build the frontend packages if you want
> to
> > >> run
> > >>the web UI.
> > >>Please make sure to read the new section, "Setting up the node /
> npm
> > >>javascript environment"
> > >> > >> CONTRIBUTING.md#setting-up-the-node--npm-javascript-
> > >> environment-only-for-www_rbac>
> > >>
> > >>in CONTRIBUTING.md
> > >>
> > >>*If you are using Apache Airflow in your production environment:*
> > >>Nothing will impact you, as every new build of Apache Airflow will
> > >> come up
> > >>with pre-built dependencies.
> > >>
> > >>Please let me know if you have any questions. Thank you
> > >>
> > >>Best,
> > >>*Verdan Mahmood*
> > >>
> > >>
> > >>On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
> > >>maximebeauche...@gmail.com> wrote:
> > >>
> > >>> Glad to see this is happening!
> > >>>
> > >>> Max
> > >>>
> > >>> On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
> > >>> ash_airflowl...@firemirror.com> wrote:
> > >>>
> >  Great! Thanks for doing this. I've left some review comments on
> > >> your PR.
> > 
> >  -ash
> > 
> > > On 9 Jul 2018, at 11:45, Verdan Mahmood <
> > >> verdan.mahm...@gmail.com>
> >  wrote:
> > >
> > > ​Hey Guys, ​
> > >
> > > In an effort to simplify the JS dependencies of Airflow
> > > ​​
> > > ,
> > > ​I've
> > > introduce
> > > ​d​
> > > npm and webpack for the package management. For now, it only
> > >> implements
> > > this in the www_rbac version of the web server.
> > > ​
> > >
> > > Pull Request: https://github.com/apache/
> > >> incubator-airflow/pull/3572
> > >
> > > The problem with the
> > > ​existing ​
> > > frontend (
> > > ​JS
> > > ) code of Airflow is that most of the custom JS is written
> > > ​with​
> > > in the html files, using the Flask's (Jinja) variables in that
> > >> JS. The
> >  next
> > > step of this effort would be to extract that custom
> > > ​JS
> > > code in separate JS files
> > > ​,​
> > > use the dependencies in those files using require or import
> > > ​ and introduce the JS automated test suite eventually. ​
> > > (At the moment, I'm simply using the CopyWebPackPlugin to copy
> > >> the
> >  required
> > > dependencies for use)
> > > ​.
> > >
> > > There are also some dependencies which are directly modified in
> > >> the
> >  codebase
> > > ​ or are outdated​
> > > . I couldn't found the
> > > ​ correct​
> > > npm versions of those libraries. (dagre-d3.js and
> > >> gantt-chart-d3v2.js).
> > > Apparently dagre-d3.js that we are using is one of the gist or
> > >> is very
> >  old
> > > version
> > > ​ not supported with webpack 4​
> > > , while the gantt-chart-d3v2 has been modified according to
> > >> Airflow's
> > > requirements
> > > ​ I believe​
> > > .
> > > ​ Used the existing libraries for now. ​
> > >
> > > ​I am currently working in a separate branch to upgrade the
> > >> DagreD3
> > > library, and updating the custom JS related to DagreD3
> > >> accordingly. ​
> > >
> > > This PR also introduces the 

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Bolke de Bruin]

After the release of 1.10 we will start removing the old “www”.

B.

> On 7 Aug 2018, at 11:03, Verdan Mahmood  wrote:
> 
> Hi Dave,
> 
> AceJS is being used only in "www" version of webserver.
> "www_rbac" does not use AceJS, and we implemented npm and webpack only for
> www_rbac.
> 
> Best,
> *Verdan Mahmood*
> 
> On Mon, Aug 6, 2018 at 11:49 PM dave.allan...@gmail.com <
> dave.allan...@gmail.com> wrote:
> 
>> Hi Verdan
>> 
>> Which version of Ace js was used? I couldn't track that
>> 
>> Cheers
>> David
>> 
>> On 2018/07/09 10:45:00, Verdan Mahmood  wrote:
>>> ​Hey Guys, ​
>>> 
>>> In an effort to simplify the JS dependencies of Airflow
>>> ​​
>>> ,
>>> ​I've
>>> introduce
>>> ​d​
>>> npm and webpack for the package management. For now, it only implements
>>> this in the www_rbac version of the web server.
>>> ​
>>> 
>>> Pull Request: https://github.com/apache/incubator-airflow/pull/3572
>>> 
>>> The problem with the
>>> ​existing ​
>>> frontend (
>>> ​JS
>>> ) code of Airflow is that most of the custom JS is written
>>> ​with​
>>> in the html files, using the Flask's (Jinja) variables in that JS. The
>> next
>>> step of this effort would be to extract that custom
>>> ​JS
>>> code in separate JS files
>>> ​,​
>>> use the dependencies in those files using require or import
>>> ​ and introduce the JS automated test suite eventually. ​
>>> (At the moment, I'm simply using the CopyWebPackPlugin to copy the
>> required
>>> dependencies for use)
>>> ​.
>>> 
>>> There are also some dependencies which are directly modified in the
>> codebase
>>> ​ or are outdated​
>>> . I couldn't found the
>>> ​ correct​
>>> npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
>>> Apparently dagre-d3.js that we are using is one of the gist or is very
>> old
>>> version
>>> ​ not supported with webpack 4​
>>> , while the gantt-chart-d3v2 has been modified according to Airflow's
>>> requirements
>>> ​ I believe​
>>> .
>>> ​ Used the existing libraries for now. ​
>>> 
>>> ​I am currently working in a separate branch to upgrade the DagreD3
>>> library, and updating the custom JS related to DagreD3 accordingly. ​
>>> 
>>> This PR also introduces the pypi_push.sh
>>> <
>> https://github.com/apache/incubator-airflow/pull/3572/files#diff-8fae684cdcc8cc8df2232c8df16f64cb
>>> 
>>> script that will generate all the JS statics before creating and
>> uploading
>>> the package.
>>> ​
>>> ​Please let me know if you guys have any questions or suggestions and I'd
>>> be happy to answer that. ​
>>> 
>>> Best,
>>> *Verdan Mahmood*
>>> (+31) 655 576 560
>>> 
>> 

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Verdan Mahmood]

Hi Dave,

AceJS is being used only in "www" version of webserver.
"www_rbac" does not use AceJS, and we implemented npm and webpack only for
www_rbac.

Best,
*Verdan Mahmood*

On Mon, Aug 6, 2018 at 11:49 PM dave.allan...@gmail.com <
dave.allan...@gmail.com> wrote:

> Hi Verdan
>
> Which version of Ace js was used? I couldn't track that
>
> Cheers
> David
>
> On 2018/07/09 10:45:00, Verdan Mahmood  wrote:
> > ​Hey Guys, ​
> >
> > In an effort to simplify the JS dependencies of Airflow
> > ​​
> > ,
> > ​I've
> >  introduce
> > ​d​
> >  npm and webpack for the package management. For now, it only implements
> > this in the www_rbac version of the web server.
> > ​
> >
> > Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> >
> > The problem with the
> > ​existing ​
> > frontend (
> > ​JS
> > ) code of Airflow is that most of the custom JS is written
> > ​with​
> > in the html files, using the Flask's (Jinja) variables in that JS. The
> next
> > step of this effort would be to extract that custom
> > ​JS
> >  code in separate JS files
> > ​,​
> > use the dependencies in those files using require or import
> > ​ and introduce the JS automated test suite eventually. ​
> > (At the moment, I'm simply using the CopyWebPackPlugin to copy the
> required
> > dependencies for use)
> > ​.
> >
> > There are also some dependencies which are directly modified in the
> codebase
> > ​ or are outdated​
> > . I couldn't found the
> > ​ correct​
> > npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> > Apparently dagre-d3.js that we are using is one of the gist or is very
> old
> > version
> > ​ not supported with webpack 4​
> > , while the gantt-chart-d3v2 has been modified according to Airflow's
> > requirements
> > ​ I believe​
> > .
> > ​ Used the existing libraries for now. ​
> >
> > ​I am currently working in a separate branch to upgrade the DagreD3
> > library, and updating the custom JS related to DagreD3 accordingly. ​
> >
> > This PR also introduces the pypi_push.sh
> > <
> https://github.com/apache/incubator-airflow/pull/3572/files#diff-8fae684cdcc8cc8df2232c8df16f64cb
> >
> > script that will generate all the JS statics before creating and
> uploading
> > the package.
> > ​
> > ​Please let me know if you guys have any questions or suggestions and I'd
> > be happy to answer that. ​
> >
> > Best,
> > *Verdan Mahmood*
> > (+31) 655 576 560
> >
>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[dave . allan . us]

Hi Verdan

Which version of Ace js was used? I couldn't track that

Cheers
David 

On 2018/07/09 10:45:00, Verdan Mahmood  wrote: 
> ​Hey Guys, ​
> 
> In an effort to simplify the JS dependencies of Airflow
> ​​
> ,
> ​I've
>  introduce
> ​d​
>  npm and webpack for the package management. For now, it only implements
> this in the www_rbac version of the web server.
> ​
> 
> Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> 
> The problem with the
> ​existing ​
> frontend (
> ​JS
> ) code of Airflow is that most of the custom JS is written
> ​with​
> in the html files, using the Flask's (Jinja) variables in that JS. The next
> step of this effort would be to extract that custom
> ​JS
>  code in separate JS files
> ​,​
> use the dependencies in those files using require or import
> ​ and introduce the JS automated test suite eventually. ​
> (At the moment, I'm simply using the CopyWebPackPlugin to copy the required
> dependencies for use)
> ​.
> 
> There are also some dependencies which are directly modified in the codebase
> ​ or are outdated​
> . I couldn't found the
> ​ correct​
> npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> Apparently dagre-d3.js that we are using is one of the gist or is very old
> version
> ​ not supported with webpack 4​
> , while the gantt-chart-d3v2 has been modified according to Airflow's
> requirements
> ​ I believe​
> .
> ​ Used the existing libraries for now. ​
> 
> ​I am currently working in a separate branch to upgrade the DagreD3
> library, and updating the custom JS related to DagreD3 accordingly. ​
> 
> This PR also introduces the pypi_push.sh
> 
> script that will generate all the JS statics before creating and uploading
> the package.
> ​
> ​Please let me know if you guys have any questions or suggestions and I'd
> be happy to answer that. ​
> 
> Best,
> *Verdan Mahmood*
> (+31) 655 576 560
> 

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Bolke de Bruin]

I think it should be removed now. 1.10.X should be the last release seri s that 
supports the old www. Do we need to vote on this?

Great work Verdan!

Verstuurd vanaf mijn iPad

> Op 23 jul. 2018 om 10:23 heeft Driesprong, Fokko  het 
> volgende geschreven:
> 
> ​Nice work Verdan.
> 
> The frontend really needed some love, thank you for picking this up. Maybe
> we should also think deprecating the old www. Keeping both of the UI's is
> something that takes a lot of time. Maybe after the release of 1.10 we can
> think of moving to Airflow 2.0, and removing the old UI.
> 
> 
> Cheers, Fokko​
> 
> 2018-07-23 10:02 GMT+02:00 Naik Kaxil :
> 
>> Awesome. Thanks @Verdan
>> 
>> On 23/07/2018, 07:58, "Verdan Mahmood"  wrote:
>> 
>>Heads-up!! This frontend change has been merged in master branch
>> recently.
>>This will impact the users working on Airflow RBAC UI only. That means:
>> 
>>*If you are a contributor/developer of Apache Airflow:*
>>You'll need to install and build the frontend packages if you want to
>> run
>>the web UI.
>>Please make sure to read the new section, "Setting up the node / npm
>>javascript environment"
>>> CONTRIBUTING.md#setting-up-the-node--npm-javascript-
>> environment-only-for-www_rbac>
>> 
>>in CONTRIBUTING.md
>> 
>>*If you are using Apache Airflow in your production environment:*
>>Nothing will impact you, as every new build of Apache Airflow will
>> come up
>>with pre-built dependencies.
>> 
>>Please let me know if you have any questions. Thank you
>> 
>>Best,
>>*Verdan Mahmood*
>> 
>> 
>>On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
>>maximebeauche...@gmail.com> wrote:
>> 
>>> Glad to see this is happening!
>>> 
>>> Max
>>> 
>>> On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
>>> ash_airflowl...@firemirror.com> wrote:
>>> 
 Great! Thanks for doing this. I've left some review comments on
>> your PR.
 
 -ash
 
> On 9 Jul 2018, at 11:45, Verdan Mahmood <
>> verdan.mahm...@gmail.com>
 wrote:
> 
> ​Hey Guys, ​
> 
> In an effort to simplify the JS dependencies of Airflow
> ​​
> ,
> ​I've
> introduce
> ​d​
> npm and webpack for the package management. For now, it only
>> implements
> this in the www_rbac version of the web server.
> ​
> 
> Pull Request: https://github.com/apache/
>> incubator-airflow/pull/3572
> 
> The problem with the
> ​existing ​
> frontend (
> ​JS
> ) code of Airflow is that most of the custom JS is written
> ​with​
> in the html files, using the Flask's (Jinja) variables in that
>> JS. The
 next
> step of this effort would be to extract that custom
> ​JS
> code in separate JS files
> ​,​
> use the dependencies in those files using require or import
> ​ and introduce the JS automated test suite eventually. ​
> (At the moment, I'm simply using the CopyWebPackPlugin to copy
>> the
 required
> dependencies for use)
> ​.
> 
> There are also some dependencies which are directly modified in
>> the
 codebase
> ​ or are outdated​
> . I couldn't found the
> ​ correct​
> npm versions of those libraries. (dagre-d3.js and
>> gantt-chart-d3v2.js).
> Apparently dagre-d3.js that we are using is one of the gist or
>> is very
 old
> version
> ​ not supported with webpack 4​
> , while the gantt-chart-d3v2 has been modified according to
>> Airflow's
> requirements
> ​ I believe​
> .
> ​ Used the existing libraries for now. ​
> 
> ​I am currently working in a separate branch to upgrade the
>> DagreD3
> library, and updating the custom JS related to DagreD3
>> accordingly. ​
> 
> This PR also introduces the pypi_push.sh
> <
 
>>> https://github.com/apache/incubator-airflow/pull/3572/files#diff-
>> 8fae684cdcc8cc8df2232c8df16f64cb
> 
> script that will generate all the JS statics before creating and
 uploading
> the package.
> ​
> ​Please let me know if you guys have any questions or
>> suggestions and
>>> I'd
> be happy to answer that. ​
> 
> Best,
> *Verdan Mahmood*
> (+31) 655 576 560
 
 
>>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Kaxil Naik
>> 
>> Data Reply
>> 2nd Floor, Nova South
>> 160 Victoria Street, Westminster
>> London SW1E 5LB - UK
>> phone: +44 (0)20 7730 6000
>> k.n...@reply.com
>> www.reply.com
>> 

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Driesprong, Fokko]

​Nice work Verdan.

The frontend really needed some love, thank you for picking this up. Maybe
we should also think deprecating the old www. Keeping both of the UI's is
something that takes a lot of time. Maybe after the release of 1.10 we can
think of moving to Airflow 2.0, and removing the old UI.


Cheers, Fokko​

2018-07-23 10:02 GMT+02:00 Naik Kaxil :

> Awesome. Thanks @Verdan
>
> On 23/07/2018, 07:58, "Verdan Mahmood"  wrote:
>
> Heads-up!! This frontend change has been merged in master branch
> recently.
> This will impact the users working on Airflow RBAC UI only. That means:
>
> *If you are a contributor/developer of Apache Airflow:*
> You'll need to install and build the frontend packages if you want to
> run
> the web UI.
> Please make sure to read the new section, "Setting up the node / npm
> javascript environment"
>  CONTRIBUTING.md#setting-up-the-node--npm-javascript-
> environment-only-for-www_rbac>
>
> in CONTRIBUTING.md
>
> *If you are using Apache Airflow in your production environment:*
> Nothing will impact you, as every new build of Apache Airflow will
> come up
> with pre-built dependencies.
>
> Please let me know if you have any questions. Thank you
>
> Best,
> *Verdan Mahmood*
>
>
> On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
> > Glad to see this is happening!
> >
> > Max
> >
> > On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
> > ash_airflowl...@firemirror.com> wrote:
> >
> > > Great! Thanks for doing this. I've left some review comments on
> your PR.
> > >
> > > -ash
> > >
> > > > On 9 Jul 2018, at 11:45, Verdan Mahmood <
> verdan.mahm...@gmail.com>
> > > wrote:
> > > >
> > > > ​Hey Guys, ​
> > > >
> > > > In an effort to simplify the JS dependencies of Airflow
> > > > ​​
> > > > ,
> > > > ​I've
> > > > introduce
> > > > ​d​
> > > > npm and webpack for the package management. For now, it only
> implements
> > > > this in the www_rbac version of the web server.
> > > > ​
> > > >
> > > > Pull Request: https://github.com/apache/
> incubator-airflow/pull/3572
> > > >
> > > > The problem with the
> > > > ​existing ​
> > > > frontend (
> > > > ​JS
> > > > ) code of Airflow is that most of the custom JS is written
> > > > ​with​
> > > > in the html files, using the Flask's (Jinja) variables in that
> JS. The
> > > next
> > > > step of this effort would be to extract that custom
> > > > ​JS
> > > > code in separate JS files
> > > > ​,​
> > > > use the dependencies in those files using require or import
> > > > ​ and introduce the JS automated test suite eventually. ​
> > > > (At the moment, I'm simply using the CopyWebPackPlugin to copy
> the
> > > required
> > > > dependencies for use)
> > > > ​.
> > > >
> > > > There are also some dependencies which are directly modified in
> the
> > > codebase
> > > > ​ or are outdated​
> > > > . I couldn't found the
> > > > ​ correct​
> > > > npm versions of those libraries. (dagre-d3.js and
> gantt-chart-d3v2.js).
> > > > Apparently dagre-d3.js that we are using is one of the gist or
> is very
> > > old
> > > > version
> > > > ​ not supported with webpack 4​
> > > > , while the gantt-chart-d3v2 has been modified according to
> Airflow's
> > > > requirements
> > > > ​ I believe​
> > > > .
> > > > ​ Used the existing libraries for now. ​
> > > >
> > > > ​I am currently working in a separate branch to upgrade the
> DagreD3
> > > > library, and updating the custom JS related to DagreD3
> accordingly. ​
> > > >
> > > > This PR also introduces the pypi_push.sh
> > > > <
> > >
> > https://github.com/apache/incubator-airflow/pull/3572/files#diff-
> 8fae684cdcc8cc8df2232c8df16f64cb
> > > >
> > > > script that will generate all the JS statics before creating and
> > > uploading
> > > > the package.
> > > > ​
> > > > ​Please let me know if you guys have any questions or
> suggestions and
> > I'd
> > > > be happy to answer that. ​
> > > >
> > > > Best,
> > > > *Verdan Mahmood*
> > > > (+31) 655 576 560
> > >
> > >
> >
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Naik Kaxil]

Awesome. Thanks @Verdan

On 23/07/2018, 07:58, "Verdan Mahmood"  wrote:

Heads-up!! This frontend change has been merged in master branch recently.
This will impact the users working on Airflow RBAC UI only. That means:

*If you are a contributor/developer of Apache Airflow:*
You'll need to install and build the frontend packages if you want to run
the web UI.
Please make sure to read the new section, "Setting up the node / npm
javascript environment"



in CONTRIBUTING.md

*If you are using Apache Airflow in your production environment:*
Nothing will impact you, as every new build of Apache Airflow will come up
with pre-built dependencies.

Please let me know if you have any questions. Thank you

Best,
*Verdan Mahmood*


On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Glad to see this is happening!
>
> Max
>
> On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
> ash_airflowl...@firemirror.com> wrote:
>
> > Great! Thanks for doing this. I've left some review comments on your PR.
> >
> > -ash
> >
> > > On 9 Jul 2018, at 11:45, Verdan Mahmood 
> > wrote:
> > >
> > > ​Hey Guys, ​
> > >
> > > In an effort to simplify the JS dependencies of Airflow
> > > ​​
> > > ,
> > > ​I've
> > > introduce
> > > ​d​
> > > npm and webpack for the package management. For now, it only 
implements
> > > this in the www_rbac version of the web server.
> > > ​
> > >
> > > Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> > >
> > > The problem with the
> > > ​existing ​
> > > frontend (
> > > ​JS
> > > ) code of Airflow is that most of the custom JS is written
> > > ​with​
> > > in the html files, using the Flask's (Jinja) variables in that JS. The
> > next
> > > step of this effort would be to extract that custom
> > > ​JS
> > > code in separate JS files
> > > ​,​
> > > use the dependencies in those files using require or import
> > > ​ and introduce the JS automated test suite eventually. ​
> > > (At the moment, I'm simply using the CopyWebPackPlugin to copy the
> > required
> > > dependencies for use)
> > > ​.
> > >
> > > There are also some dependencies which are directly modified in the
> > codebase
> > > ​ or are outdated​
> > > . I couldn't found the
> > > ​ correct​
> > > npm versions of those libraries. (dagre-d3.js and 
gantt-chart-d3v2.js).
> > > Apparently dagre-d3.js that we are using is one of the gist or is very
> > old
> > > version
> > > ​ not supported with webpack 4​
> > > , while the gantt-chart-d3v2 has been modified according to Airflow's
> > > requirements
> > > ​ I believe​
> > > .
> > > ​ Used the existing libraries for now. ​
> > >
> > > ​I am currently working in a separate branch to upgrade the DagreD3
> > > library, and updating the custom JS related to DagreD3 accordingly. ​
> > >
> > > This PR also introduces the pypi_push.sh
> > > <
> >
> 
https://github.com/apache/incubator-airflow/pull/3572/files#diff-8fae684cdcc8cc8df2232c8df16f64cb
> > >
> > > script that will generate all the JS statics before creating and
> > uploading
> > > the package.
> > > ​
> > > ​Please let me know if you guys have any questions or suggestions and
> I'd
> > > be happy to answer that. ​
> > >
> > > Best,
> > > *Verdan Mahmood*
> > > (+31) 655 576 560
> >
> >
>






Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Verdan Mahmood]

Heads-up!! This frontend change has been merged in master branch recently.
This will impact the users working on Airflow RBAC UI only. That means:

*If you are a contributor/developer of Apache Airflow:*
You'll need to install and build the frontend packages if you want to run
the web UI.
Please make sure to read the new section, "Setting up the node / npm
javascript environment"


in CONTRIBUTING.md

*If you are using Apache Airflow in your production environment:*
Nothing will impact you, as every new build of Apache Airflow will come up
with pre-built dependencies.

Please let me know if you have any questions. Thank you

Best,
*Verdan Mahmood*


On Sun, Jul 15, 2018 at 6:52 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Glad to see this is happening!
>
> Max
>
> On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
> ash_airflowl...@firemirror.com> wrote:
>
> > Great! Thanks for doing this. I've left some review comments on your PR.
> >
> > -ash
> >
> > > On 9 Jul 2018, at 11:45, Verdan Mahmood 
> > wrote:
> > >
> > > ​Hey Guys, ​
> > >
> > > In an effort to simplify the JS dependencies of Airflow
> > > ​​
> > > ,
> > > ​I've
> > > introduce
> > > ​d​
> > > npm and webpack for the package management. For now, it only implements
> > > this in the www_rbac version of the web server.
> > > ​
> > >
> > > Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> > >
> > > The problem with the
> > > ​existing ​
> > > frontend (
> > > ​JS
> > > ) code of Airflow is that most of the custom JS is written
> > > ​with​
> > > in the html files, using the Flask's (Jinja) variables in that JS. The
> > next
> > > step of this effort would be to extract that custom
> > > ​JS
> > > code in separate JS files
> > > ​,​
> > > use the dependencies in those files using require or import
> > > ​ and introduce the JS automated test suite eventually. ​
> > > (At the moment, I'm simply using the CopyWebPackPlugin to copy the
> > required
> > > dependencies for use)
> > > ​.
> > >
> > > There are also some dependencies which are directly modified in the
> > codebase
> > > ​ or are outdated​
> > > . I couldn't found the
> > > ​ correct​
> > > npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> > > Apparently dagre-d3.js that we are using is one of the gist or is very
> > old
> > > version
> > > ​ not supported with webpack 4​
> > > , while the gantt-chart-d3v2 has been modified according to Airflow's
> > > requirements
> > > ​ I believe​
> > > .
> > > ​ Used the existing libraries for now. ​
> > >
> > > ​I am currently working in a separate branch to upgrade the DagreD3
> > > library, and updating the custom JS related to DagreD3 accordingly. ​
> > >
> > > This PR also introduces the pypi_push.sh
> > > <
> >
> https://github.com/apache/incubator-airflow/pull/3572/files#diff-8fae684cdcc8cc8df2232c8df16f64cb
> > >
> > > script that will generate all the JS statics before creating and
> > uploading
> > > the package.
> > > ​
> > > ​Please let me know if you guys have any questions or suggestions and
> I'd
> > > be happy to answer that. ​
> > >
> > > Best,
> > > *Verdan Mahmood*
> > > (+31) 655 576 560
> >
> >
>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Maxime Beauchemin]

Glad to see this is happening!

Max

On Mon, Jul 9, 2018 at 6:37 AM Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

> Great! Thanks for doing this. I've left some review comments on your PR.
>
> -ash
>
> > On 9 Jul 2018, at 11:45, Verdan Mahmood 
> wrote:
> >
> > ​Hey Guys, ​
> >
> > In an effort to simplify the JS dependencies of Airflow
> > ​​
> > ,
> > ​I've
> > introduce
> > ​d​
> > npm and webpack for the package management. For now, it only implements
> > this in the www_rbac version of the web server.
> > ​
> >
> > Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> >
> > The problem with the
> > ​existing ​
> > frontend (
> > ​JS
> > ) code of Airflow is that most of the custom JS is written
> > ​with​
> > in the html files, using the Flask's (Jinja) variables in that JS. The
> next
> > step of this effort would be to extract that custom
> > ​JS
> > code in separate JS files
> > ​,​
> > use the dependencies in those files using require or import
> > ​ and introduce the JS automated test suite eventually. ​
> > (At the moment, I'm simply using the CopyWebPackPlugin to copy the
> required
> > dependencies for use)
> > ​.
> >
> > There are also some dependencies which are directly modified in the
> codebase
> > ​ or are outdated​
> > . I couldn't found the
> > ​ correct​
> > npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> > Apparently dagre-d3.js that we are using is one of the gist or is very
> old
> > version
> > ​ not supported with webpack 4​
> > , while the gantt-chart-d3v2 has been modified according to Airflow's
> > requirements
> > ​ I believe​
> > .
> > ​ Used the existing libraries for now. ​
> >
> > ​I am currently working in a separate branch to upgrade the DagreD3
> > library, and updating the custom JS related to DagreD3 accordingly. ​
> >
> > This PR also introduces the pypi_push.sh
> > <
> https://github.com/apache/incubator-airflow/pull/3572/files#diff-8fae684cdcc8cc8df2232c8df16f64cb
> >
> > script that will generate all the JS statics before creating and
> uploading
> > the package.
> > ​
> > ​Please let me know if you guys have any questions or suggestions and I'd
> > be happy to answer that. ​
> >
> > Best,
> > *Verdan Mahmood*
> > (+31) 655 576 560
>
>

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

[Ash Berlin-Taylor]

Great! Thanks for doing this. I've left some review comments on your PR.

-ash

> On 9 Jul 2018, at 11:45, Verdan Mahmood  wrote:
> 
> ​Hey Guys, ​
> 
> In an effort to simplify the JS dependencies of Airflow
> ​​
> ,
> ​I've
> introduce
> ​d​
> npm and webpack for the package management. For now, it only implements
> this in the www_rbac version of the web server.
> ​
> 
> Pull Request: https://github.com/apache/incubator-airflow/pull/3572
> 
> The problem with the
> ​existing ​
> frontend (
> ​JS
> ) code of Airflow is that most of the custom JS is written
> ​with​
> in the html files, using the Flask's (Jinja) variables in that JS. The next
> step of this effort would be to extract that custom
> ​JS
> code in separate JS files
> ​,​
> use the dependencies in those files using require or import
> ​ and introduce the JS automated test suite eventually. ​
> (At the moment, I'm simply using the CopyWebPackPlugin to copy the required
> dependencies for use)
> ​.
> 
> There are also some dependencies which are directly modified in the codebase
> ​ or are outdated​
> . I couldn't found the
> ​ correct​
> npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
> Apparently dagre-d3.js that we are using is one of the gist or is very old
> version
> ​ not supported with webpack 4​
> , while the gantt-chart-d3v2 has been modified according to Airflow's
> requirements
> ​ I believe​
> .
> ​ Used the existing libraries for now. ​
> 
> ​I am currently working in a separate branch to upgrade the DagreD3
> library, and updating the custom JS related to DagreD3 accordingly. ​
> 
> This PR also introduces the pypi_push.sh
> 
> script that will generate all the JS statics before creating and uploading
> the package.
> ​
> ​Please let me know if you guys have any questions or suggestions and I'd
> be happy to answer that. ​
> 
> Best,
> *Verdan Mahmood*
> (+31) 655 576 560

Airflow's JS code (and dependencies) manageable via npm and webpack

[Verdan Mahmood]

​Hey Guys, ​

In an effort to simplify the JS dependencies of Airflow
​​
,
​I've
 introduce
​d​
 npm and webpack for the package management. For now, it only implements
this in the www_rbac version of the web server.
​

Pull Request: https://github.com/apache/incubator-airflow/pull/3572

The problem with the
​existing ​
frontend (
​JS
) code of Airflow is that most of the custom JS is written
​with​
in the html files, using the Flask's (Jinja) variables in that JS. The next
step of this effort would be to extract that custom
​JS
 code in separate JS files
​,​
use the dependencies in those files using require or import
​ and introduce the JS automated test suite eventually. ​
(At the moment, I'm simply using the CopyWebPackPlugin to copy the required
dependencies for use)
​.

There are also some dependencies which are directly modified in the codebase
​ or are outdated​
. I couldn't found the
​ correct​
npm versions of those libraries. (dagre-d3.js and gantt-chart-d3v2.js).
Apparently dagre-d3.js that we are using is one of the gist or is very old
version
​ not supported with webpack 4​
, while the gantt-chart-d3v2 has been modified according to Airflow's
requirements
​ I believe​
.
​ Used the existing libraries for now. ​

​I am currently working in a separate branch to upgrade the DagreD3
library, and updating the custom JS related to DagreD3 accordingly. ​

This PR also introduces the pypi_push.sh

script that will generate all the JS statics before creating and uploading
the package.
​
​Please let me know if you guys have any questions or suggestions and I'd
be happy to answer that. ​

Best,
*Verdan Mahmood*
(+31) 655 576 560