Hi Max, The canonical distribution would be what we have in git right now (ie. before running python sdist). The rest is just convenience packages. So npm, would solve the issue as long as we don’t rely on any non APL compatible packages in core. I don’t think npm/yarn/webpack needs to be done for 1.8.3, but considering the messy javascript that we currently have it would be nice to put it on the todo.
Cheers Bolke > On 9 Jul 2017, at 06:46, Maxime Beauchemin <[email protected]> wrote: > > As far as I understand npm would not solve the problem as typically we'd > build our "entry" files and distribute that with Airflow as static files. > Those entry files would contain these other npm packages, minified. (from > my understanding that is the same issue as packaging the libs themselves) > > To make them runtime deps would be atypical and more complicated. `airflow > webserver` would need to "build" (npm install/webpack) and the webserver > would have to serve these static files out of some that temp location > (perhaps ~/.airflow/airflow.entry.js) as opposed to out of `site-packages`. > > Also note that Airflow's javascript is in pretty bad shape (scattered in > jinja templates files) and it would take quite a significant amount of work > to move to using npm/webpack. > > I'm back from vacation and will have things to catch up on next week but > I'll try to find time to look into some of this. > > On Thu, Jul 6, 2017 at 1:10 PM, Bolke de Bruin <[email protected] > <mailto:[email protected]>> wrote: > >> Hi Folks, >> >> We probably need to adjust our release process as can be observed in the >> IPMC thread. As we are packaging a “sdist” it does not pass license checks >> and one cannot verify the validity of what we are doing. It was suggested >> by one of the maintainers of another python project to create 3 different >> packages: >> >> 1. A source tarball which is essentially a snapshot of the repository >> 2. A sdist >> 3. A bdist >> >> 1 should then be the canonical Apache release. It should be accompanied by >> build instructions and it should pass RAT checks. This is the package we >> will vote on. >> 2 is what we have voted upon until now. It should contain (it does) >> LICENSE, NOTICE, and DISCLAIMER >> 3 bdist, wheel package. Same as 2. Not really required, but more a >> convenience package as is 2 >> >> 2 and 3 can be published to PyPi. >> >> Max: can you take care of this? We need to vote on 1. Build instruction >> could be added to an INSTALL file or just to the README.md file? See for >> inspiration the GitHub page of ariatosca: https://github.com/apache/ >> incubator-ariatosca <https://github.com/apache/incubator-ariatosca >> <https://github.com/apache/incubator-ariatosca>> >> >> In the meantime I am figuring out an issue with one of the dependencies of >> nvd3 which might be or have been GPL3 which is incompatible with the APL, >> which we are distributing together with our source. Ideally, we should move >> to a “yarn/webpack” build which will resolve those issues automatically as >> these become runtime dependencies then in case of 1,2,3. >> >> Cheers >> Bolke >> >> >> >>> On 6 Jul 2017, at 05:20, Sumit Maheshwari <[email protected] >>> <mailto:[email protected]>> >> wrote: >>> >>> Awesome.. thanks a lot Max for taking the RM responsibility.. >>> >>> On Jul 5, 2017 11:10 PM, "Chris Riccomini" <[email protected] >>> <mailto:[email protected]>> >> wrote: >>> >>>> Nice, thanks Max! >>>> >>>> On Mon, Jul 3, 2017 at 2:36 PM, Maxime Beauchemin < >>>> [email protected] <mailto:[email protected]>> wrote: >>>> >>>>> Hello, >>>>> >>>>> Apache Airflow (incubating) 1.8.2 (RC2) has been accepted. >>>>> >>>>> 5 “+1” votes received, 4 out of which are "binding": >>>>> >>>>> - Maxime Beauchemin (binding) >>>>> - Bolke de Bruin (binding) >>>>> - Chris Riccomini (binding) >>>>> - Sumit Maheshwari (binding) >>>>> >>>>> - Kengo Seki (non-binding) >>>>> >>>>> Vote thread (start): >>>>> *http://mail-archives.apache.org/mod_mbox/incubator- >>>>> <http://mail-archives.apache.org/mod_mbox/incubator-> >>>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc-- >>>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com <http://40mail.gmail.com/>%3E >>>>> <http://mail-archives.apache.org/mod_mbox/incubator- >>>>> <http://mail-archives.apache.org/mod_mbox/incubator-> >>>>> airflow-dev/201706.mbox/%3CCAHEEp7XRRk__hGUMjOCw%2BTW0Gc-- >>>>> 78qEewcsAsQBLrv1x6j71w%40mail.gmail.com <http://40mail.gmail.com/>%3E>* >>>>> >>>>> Next steps: >>>>> 1) will start the voting process at the IPMC mailinglist. I don’t >> expect >>>>> changes. >>>>> 2) Only after the positive voting on the IPMC and finalisation I will >>>>> rebrand the RC to Release. >>>>> 3) I will upload it to the incubator release page, then the tar ball >>>> needs >>>>> to propagate to the mirrors. >>>>> 4) Update the website >>>>> 5) Finally, will upload to Pypi. >>>>> >>>>> Cheers, >>>>> >>>>> Max >>>>> >>>>> On Mon, Jul 3, 2017 at 12:11 PM, Bolke de Bruin <[email protected] >>>>> <mailto:[email protected]>> >>>> wrote: >>>>> >>>>>> Ping? >>>>>> >>>>>> >>>>>>> On 30 Jun 2017, at 14:21, Bolke de Bruin <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Max, >>>>>>> >>>>>>> I think you can close the vote? >>>>>>> >>>>>>> Bolke >>>>>>> >>>>>>>> On 27 Jun 2017, at 02:45, Kengo Seki <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> +1 (non-binding) >>>>>>>> >>>>>>>> - verified signatures and checksums >>>>>>>> - ran scheduler and webserver, confirmed they worked fine >>>>>>>> - confirmed the latest fix on v1.8 branch (AIRFLOW-809) is included >>>>>>>> >>>>>>>> Kengo Seki <[email protected] <mailto:[email protected]>> >>>>>>>> >>>>>>>> >>>>>>>> 2017-06-27 8:53 GMT+09:00 Chris Riccomini <[email protected] >>>>>>>> <mailto:[email protected]>>: >>>>>>>>> +1 (binding) >>>>>>>>> >>>>>>>>> Been running in our dev env, and everything looks good. >>>>>>>>> >>>>>>>>> On Mon, Jun 26, 2017 at 3:00 PM, Alex Guziel < >>>> [email protected] <mailto:[email protected]> >>>>> . >>>>>> invalid >>>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yeah that makes sense. It pages by default at 500 so it explains >>>> why >>>>>> we saw >>>>>>>>>> it. >>>>>>>>>> >>>>>>>>>> On Mon, Jun 26, 2017 at 2:53 PM, Chris Riccomini < >>>>>> [email protected] <mailto:[email protected]>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> In 1.8.1, the "DAGs" page has "Show <blank> entries". In 1.8.2, >>>> it >>>>>> has >>>>>>>>>>> "Show <25> entries". So it looks like prior to 1.8.2, the >>>>> pagination >>>>>> was >>>>>>>>>>> broken in the sense that it defaulted to the whole list. We have >>>>> 479 >>>>>> DAGs >>>>>>>>>>> in one env, and it shows them all. It looks like someone fixed >>>> the >>>>>> entry >>>>>>>>>> to >>>>>>>>>>> default to 25 now, which exposed the problem for our >>>> environments. >>>>>>>>>>> >>>>>>>>>>> On Mon, Jun 26, 2017 at 2:47 PM, Alex Guziel < >>>>> [email protected] <mailto:[email protected]> >>>>>> . >>>>>>>>>>> invalid >>>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> We're running 1.8.0 + some extras, and none of us added >>>> pagination >>>>>>>>>>>> recently, and our homepage is paginated. Are you sure it's not >>>> the >>>>>>>>>> number >>>>>>>>>>>> of dags crossing the threshold? Maybe it's some Flask version >>>>> thing? >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:45 PM, Chris Riccomini < >>>>>>>>>> [email protected] <mailto:[email protected]>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes, I did the 1.8.1 release. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:44 PM, Alex Guziel < >>>>>> [email protected] <mailto:[email protected]> >>>>>>>>>> . >>>>>>>>>>>>> invalid >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> There's no pagination in 1.8.1? Are you sure? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:37 PM, Chris Riccomini < >>>>>>>>>>>> [email protected] <mailto:[email protected]>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> It's not happening on 1.8.1 (since there's no pagination in >>>>> that >>>>>>>>>>>>>> version), >>>>>>>>>>>>>>> so I'd count this as a regression. I wouldn't say it's >>>>> blocking, >>>>>>>>>>> but >>>>>>>>>>>>> it's >>>>>>>>>>>>>>> pretty ugly. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:34 PM, Alex Guziel < >>>>>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>>>>>> . >>>>>>>>>>>>>>> invalid >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm not so sure this is a new issue. I think we've seen it >>>> on >>>>>>>>>> our >>>>>>>>>>>>>>>> production for quite a while. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 2:31 PM, Chris Riccomini < >>>>>>>>>>>>>> [email protected] <mailto:[email protected]>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am seeing a strange UI behavior on 1.8.2.RC2. I've >>>> opened a >>>>>>>>>>>> JIRA >>>>>>>>>>>>>>> here: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1348 >>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1348> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Has anyone else seen this? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:27 AM, Sumit Maheshwari < >>>>>>>>>>>>>>>> [email protected] <mailto:[email protected]>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> +1, binding. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Jun 26, 2017 at 3:49 PM, Bolke de Bruin < >>>>>>>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> We have been running it for the last couple of days. No >>>>>>>>>>>> issues >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> seems >>>>>>>>>>>>>>>>>>> more responsive. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> +1, binding >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Bolke >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 25 Jun 2017, at 01:10, Maxime Beauchemin < >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Dear all, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is baked and available at: >>>>>>>>>>>>>>>>>>>> https://dist.apache.org/repos/ >>>>>>>>>>>>>>>>>>>> <https://dist.apache.org/repos/> >>>>>>>>>> dist/dev/incubator/airflow >>>>>>>>>>> , >>>>>>>>>>>>>> public >>>>>>>>>>>>>>>>> keys >>>>>>>>>>>>>>>>>>>> are available >>>>>>>>>>>>>>>>>>>> at https://dist.apache.org/repos/ >>>>>>>>>>>>>>>>>>>> <https://dist.apache.org/repos/> >>>>>>>>>>>>>> dist/release/incubator/airflow. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Note that RC1 was the first RC (skipped RC0) and was >>>>>>>>>>> never >>>>>>>>>>>>>>>> announced >>>>>>>>>>>>>>>>>>> since >>>>>>>>>>>>>>>>>>>> it had issues coming out of the oven, so RC2 is the >>>>>>>>>> first >>>>>>>>>>>>>> public >>>>>>>>>>>>>>>> RC. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 1.8.2 RC2 is build on to of 1.8.1 with these listed >>>>>>>>>>>>> "cherries" >>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>> top. >>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>> added the JIRAs that were identified blockers and >>>>>>>>>>> targeted >>>>>>>>>>>>>>> 1.8.2. I >>>>>>>>>>>>>>>>>>>> attempted to bring in all of the JIRAs that targeted >>>>>>>>>>> 1.8.2 >>>>>>>>>>>>> but >>>>>>>>>>>>>>>> bailed >>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>> the ones that were generating merge conflicts. I also >>>>>>>>>>> added >>>>>>>>>>>>> all >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> JIRAs that we've been running in production at Airbnb. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Issues fixed: >>>>>>>>>>>>>>>>>>>> 9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ >>>>>>>>>>> ColumnOperator >>>>>>>>>>>>> When >>>>>>>>>>>>>>>>> Testing >>>>>>>>>>>>>>>>>>>> Booleans >>>>>>>>>>>>>>>>>>>> 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all >>>>>>>>>>> downstream >>>>>>>>>>>>>> tasks >>>>>>>>>>>>>>>>>>>> 93825d5 [AIRFLOW-XXX] Re-enable caching for hadoop >>>>>>>>>>>> components >>>>>>>>>>>>>>>>>>>> 33a9dcb [AIRFLOW-XXX] Pin Hive and Hadoop to a specific >>>>>>>>>>>>> version >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>>> writable warehouse dir >>>>>>>>>>>>>>>>>>>> 7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask >>>>>>>>>>>>>>>>>>>> 570b2ed [AIRFLOW-1294] Backfills can loose tasks to >>>>>>>>>>> execute >>>>>>>>>>>>>>>>>>>> 3f48d48 [AIRFLOW-1291] Update NOTICE and LICENSE files >>>>>>>>>> to >>>>>>>>>>>>> match >>>>>>>>>>>>>>> ASF >>>>>>>>>>>>>>>>>>>> requirements >>>>>>>>>>>>>>>>>>>> 69bd269 [AIRFLOW-1160] Update Spark parameters for >>>>>>>>>> Mesos >>>>>>>>>>>>>>>>>>>> 9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for custom >>>>>>>>>>>> filters >>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> Jinja2 >>>>>>>>>>>>>>>>>>>> templates >>>>>>>>>>>>>>>>>>>> 6de5330 [AIRFLOW-1119] Fix unload query so headers are >>>>>>>>>> on >>>>>>>>>>>>> first >>>>>>>>>>>>>>>> row[] >>>>>>>>>>>>>>>>>>>> b4e9eb8 [AIRFLOW-1089] Add Spark application arguments >>>>>>>>>>>>>>>>>>>> a4083f3 [AIRFLOW-1078] Fix latest_runs endpoint for old >>>>>>>>>>>> flask >>>>>>>>>>>>>>>>> versions >>>>>>>>>>>>>>>>>>>> 7a02841 [AIRFLOW-1074] Don't count queued tasks for >>>>>>>>>>>>> concurrency >>>>>>>>>>>>>>>>> limits >>>>>>>>>>>>>>>>>>>> a2c18a5 [AIRFLOW-1064] Change default sort to job_id >>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> TaskInstanceModelView >>>>>>>>>>>>>>>>>>>> d1c64ab [AIRFLOW-1038] Specify celery serialization >>>>>>>>>>> options >>>>>>>>>>>>>>>>> explicitly >>>>>>>>>>>>>>>>>>>> b4ee88a [AIRFLOW-1036] Randomize exponential backoff >>>>>>>>>>>>>>>>>>>> 9fca409 [AIRFLOW-993] Update date inference logic >>>>>>>>>>>>>>>>>>>> 272c2f5 [AIRFLOW-1167] Support microseconds in FTPHook >>>>>>>>>>>>>>> modification >>>>>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>>>>>> c7c0b72 [AIRFLOW-1179] Fix Pandas 0.2x breaking Google >>>>>>>>>>>>> BigQuery >>>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>> acd0166 [AIRFLOW-1263] Dynamic height for charts >>>>>>>>>>>>>>>>>>>> 7f33f6e [AIRFLOW-1266] Increase width of gantt y axis >>>>>>>>>>>>>>>>>>>> fc33c04 [AIRFLOW-1290] set docs author to 'Apache >>>>>>>>>>> Airflow' >>>>>>>>>>>>>>>>>>>> 2e9eee3 [AIRFLOW-1282] Fix known event column sorting >>>>>>>>>>>>>>>>>>>> 2389a8a [AIRFLOW-1166] Speed up >>>>>>>>>>>>> _change_state_for_tis_without_ >>>>>>>>>>>>>>>> dagrun >>>>>>>>>>>>>>>>>>>> bf966e6 [AIRFLOW-1192] Some enhancements to >>>>>>>>>>> qubole_operator >>>>>>>>>>>>>>>>>>>> 57d5bcd [AIRFLOW-1281] Sort variables by key field by >>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>> 802fc15 [AIRFLOW-1244] Forbid creation of a pool with >>>>>>>>>>> empty >>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>> 1232b6a [AIRFLOW-1243] DAGs table has no default >>>>>>>>>> entries >>>>>>>>>>> to >>>>>>>>>>>>>> show >>>>>>>>>>>>>>>>>>>> b0ba3c9 [AIRFLOW-1227] Remove empty column on the Logs >>>>>>>>>>> view >>>>>>>>>>>>>>>>>>>> c406652 [AIRFLOW-1226] Remove empty column on the Jobs >>>>>>>>>>> view >>>>>>>>>>>>>>>>>>>> 51a83cc [AIRFLOW-1199] Fix create modal >>>>>>>>>>>>>>>>>>>> cac7d4c [AIRFLOW-1200] Forbid creation of a variable >>>>>>>>>> with >>>>>>>>>>>> an >>>>>>>>>>>>>>> empty >>>>>>>>>>>>>>>>> key >>>>>>>>>>>>>>>>>>>> 5f3ee52 [AIRFLOW-1186] Sort dag.get_task_instances by >>>>>>>>>>>>>>>> execution_date >>>>>>>>>>>>>>>>>>>> f446c08 [AIRFLOW-1145] Fix closest_date_partition >>>>>>>>>>> function >>>>>>>>>>>>> with >>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>> to True If we're looking for the closest date before, >>>>>>>>>> we >>>>>>>>>>>>> should >>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> latest date in the list of date before. >>>>>>>>>>>>>>>>>>>> 93b8e96 [AIRFLOW-1180] Fix flask-wtf version for >>>>>>>>>>>>>>>> test_csrf_rejection >>>>>>>>>>>>>>>>>>>> bb56805 [AIRFLOW-1170] DbApiHook insert_rows inserts >>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>> separately >>>>>>>>>>>>>>>>>>>> 093b2f0 [AIRFLOW-1150] Fix scripts execution in >>>>>>>>>> sparksql >>>>>>>>>>>>> hook[] >>>>>>>>>>>>>>>>>>>> 777f181 [AIRFLOW-1168] Add closing() to all connections >>>>>>>>>>> and >>>>>>>>>>>>>>> cursors >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Max
