Hi max, PR 2365 is merged into master and also into v1-8-test.
I have added: https://issues.apache.org/jira/browse/AIRFLOW-935 <https://issues.apache.org/jira/browse/AIRFLOW-935> and https://issues.apache.org/jira/browse/AIRFLOW-860 <https://issues.apache.org/jira/browse/AIRFLOW-860> To the list fro 1.8.2. The PR for this has been in master for some time: https://github.com/apache/incubator-airflow/pull/2120 <https://github.com/apache/incubator-airflow/pull/2120> Can you consider it for 1.8.2? Basically I promised to push to the matter :-). No blockers anymore as far as I know for 1.8.2. Bolke P.S. A friendly request if any of the committers do +1 or a LGTM to a PR, please also merge the PR instead of leaving it to others. “airflow-pr merge XXX” is there to help. > On 16 Jun 2017, at 15:11, Bolke de Bruin <bdbr...@gmail.com> wrote: > > It is now pinned. I think Cloudera messed up their release for some reason. > > Builds are succeeding again. > > Please note I have one blocker out for 1.8.2: > https://github.com/apache/incubator-airflow/pull/2365 > <https://github.com/apache/incubator-airflow/pull/2365> > > Bolke > >> On 15 Jun 2017, at 16:43, Maxime Beauchemin <maximebeauche...@gmail.com >> <mailto:maximebeauche...@gmail.com>> wrote: >> >> Awesome, thanks for looking into it. Is there a way we can pin the Cloudera >> tarball to avoid surprises like this? >> >> On Thu, Jun 15, 2017 at 7:17 AM, Bolke de Bruin <bdbr...@gmail.com >> <mailto:bdbr...@gmail.com>> wrote: >> >>> Found the issue for the permission denied part: beeline is not executable >>> in the latest tarball from Cloudera. I’ll have a workaround for this in a >>> few minutes and will push it to Apache right away. >>> >>> Bolke. >>> >>>> On 14 Jun 2017, at 18:15, Bolke de Bruin <bdbr...@gmail.com >>>> <mailto:bdbr...@gmail.com>> wrote: >>>> >>>> Hi Max, >>>> >>>> I have created https://github.com/apache/incubator-airflow/pull/2365 >>>> <https://github.com/apache/incubator-airflow/pull/2365> < >>> https://github.com/apache/incubator-airflow/pull/2365 >>> <https://github.com/apache/incubator-airflow/pull/2365>> for AIRFLOW-1296, >>> which I think should be treated as a blocker. >>>> >>>> I think Travis’ is failing due to a dependency upgrade of Dask. >>>> >>>> The permission denied error seems to come from a new Travis config that >>> does not allow the creation of “/user/hive/warehouse” by the normal user. >>> Probably a “sudo mkdir “ and a “sudo chown” will help here. Let me check. >>>> >>>> Bolke >>>> >>>>> On 14 Jun 2017, at 00:26, Maxime Beauchemin <maximebeauche...@gmail.com >>>>> <mailto:maximebeauche...@gmail.com> >>> <mailto:maximebeauche...@gmail.com <mailto:maximebeauche...@gmail.com>>> >>> wrote: >>>>> >>>>> A quick update on my progress. >>>>> >>>>> I ran through the "Releasing Airflow" wiki playbook only to realize I >>> had >>>>> published a version that failed the Travis build >>>>> <https://travis-ci.org/mistercrunch/incubator-airflow/jobs/242532965 >>>>> <https://travis-ci.org/mistercrunch/incubator-airflow/jobs/242532965> < >>> https://travis-ci.org/mistercrunch/incubator-airflow/jobs/242532965 >>> <https://travis-ci.org/mistercrunch/incubator-airflow/jobs/242532965>>> >>>>> afterwards. I've >>>>> been updating the wiki as I go and it seems like it will be better the >>> next >>>>> time around. I'm getting "Permission denied" on the Hive hook >>> subprocess if >>>>> anyone can shed some light on that. >>>>> >>>>> So I need to fix the unit tests at this point, and probably go straight >>> to >>>>> `rc2` to avoid changing the files out there for `rc1`. >>>>> >>>>> Max >>>>> >>>>> On Thu, Jun 8, 2017 at 4:54 PM, Ruslan Dautkhanov <dautkha...@gmail.com >>>>> <mailto:dautkha...@gmail.com> >>> <mailto:dautkha...@gmail.com <mailto:dautkha...@gmail.com>>> >>>>> wrote: >>>>> >>>>>> It would be great if somebody would have a look at following 3 jiras >>>>>> >>>>>> I've flagged >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1013 >>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1013> < >>> https://issues.apache.org/jira/browse/AIRFLOW-1013 >>> <https://issues.apache.org/jira/browse/AIRFLOW-1013>> >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1178 >>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1178> >>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1055 >>>>>> >>>>>> Two of them were Blockers for one of previous 1.8 releases but were >>>>>> de-escalated because no resources to fix them. >>>>>> >>>>>> I'm currently Assignee on AIRFLOW-1055 - can't unassign myself - feel >>> free >>>>>> to scratch my name there. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ruslan Dautkhanov >>>>>> >>>>>> On Fri, Jun 9, 2017 at 12:23 AM, Maxime Beauchemin < >>>>>> maximebeauche...@gmail.com <mailto:maximebeauche...@gmail.com>> wrote: >>>>>> >>>>>>> Cool. I'll let everyone know when RC1 is out. >>>>>>> >>>>>>> @community, if you think anything else should be a blocker for 1.8.2 >>>>>> please >>>>>>> flag it in Jira very soon or it will miss the release train! >>>>>>> >>>>>>> Max >>>>>>> >>>>>>> On Thu, Jun 8, 2017 at 9:06 AM, Chris Riccomini < >>> criccom...@apache.org <mailto:criccom...@apache.org>> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for doing all this, Max! When you're ready, I can deploy to >>> some >>>>>>> of >>>>>>>> our environments to verify. Just let me know. >>>>>>>> >>>>>>>> On Thu, Jun 8, 2017 at 8:42 AM, Maxime Beauchemin < >>>>>>>> maximebeauche...@gmail.com <mailto:maximebeauche...@gmail.com>> wrote: >>>>>>>> >>>>>>>>> I branched off `v1-8-test` so all should be good. I just didn't know >>>>>>> if I >>>>>>>>> could move ahead with that branch just yet so I branched off. I just >>>>>>> got >>>>>>>>> back on `v1-8-test` and pushed what I have to Apache [somehow I had >>>>>> to >>>>>>>>> rebase meaning someone added something over the past 24 hours]. >>>>>>>>> >>>>>>>>> I just set `Fix Version` of AIRFLOW-1294 to 1.8.2 and set it to >>>>>>> blocker. >>>>>>>>> >>>>>>>>> I cherry-picked all the commits targeting 1.8.2 I could without >>>>>> getting >>>>>>>>> into conflicts. I'd only work at resolving conflicts on blockers. My >>>>>>> plan >>>>>>>>> is to add only the 2 rows in red for RC1. >>>>>>>>> >>>>>>>>> Here's the current output of `airflow-jira 1.8.2`: >>>>>>>>> >>>>>>>>> ISSUE ID |TYPE ||PRIORITY ||STATUS |DESCRIPTION >>>>>>>>> |MERGED|PR |COMMITAIRFLOW-1294 >>>>>>>>> |Bug ||Blocker ||Open |Backfills can loose tasks to >>>>>>>>> execute due to tasks |0 |- |- >>>>>>>>> AIRFLOW-1291 |Bug ||Blocker ||Open |Update >>>>>>>>> NOTICE and LICENSE files to meet ASF specs |0 |- |- >>>>>>>>> AIRFLOW-1290 |Bug ||Major ||Resolved |Change docs >>>>>>>>> author from "Maxime Beauchemin" to "Ap|1 |#na >>>>>>>>> |fc33c040605e7b0121739af841790336bdfa3948 >>>>>>>>> AIRFLOW-1282 |Bug ||Major ||Resolved |Fix known >>>>>>>>> event column sorting |1 |#2350 >>>>>>>>> |2e9eee3697f6261de61c77ba584ace14b22751f0 >>>>>>>>> AIRFLOW-1281 |Bug ||Minor ||Resolved |Sort >>>>>> variables >>>>>>>>> by key field by default |1 |#2347 >>>>>>>>> |57d5bcdad4202fd8a94be07b4aeacee065801481 >>>>>>>>> AIRFLOW-1275 |Bug ||Major ||In Progress|Fix >>> `airflow >>>>>>>>> pool` command exception |0 |- |- >>>>>>>>> AIRFLOW-1274 |Bug ||Major ||Open |HttpSensor >>>>>>>>> parameter params is overriding BaseOper|0 |- |- >>>>>>>>> AIRFLOW-1266 |Bug ||Major ||Open |Long task >>>>>>>>> names are truncated in gannt view |1 |#2345 >>>>>>>>> |7f33f6e2fe4149e547860fdb237da26a5b7023ec >>>>>>>>> AIRFLOW-1263 |Bug ||Major ||Resolved |Airflow >>>>>> Charts >>>>>>>>> Pages Should Have Dynamic Heights |1 |#2344 >>>>>>>>> |acd0166d8c7f5639898791848d985f6441b3b5b5 >>>>>>>>> AIRFLOW-1244 |Bug ||Major ||Resolved |Forbid >>>>>>>>> creation of a pool with empty name |1 |#2324 >>>>>>>>> |802fc1549d0f173c646794f2476156d34383adc0 >>>>>>>>> AIRFLOW-1243 |Bug ||Minor ||Resolved |DAGs table >>>>>> has >>>>>>>>> no default entries to show |1 |#2323 >>>>>>>>> |1232b6a4fd691e0faf081d4cf792a6d96d33b595 >>>>>>>>> AIRFLOW-1227 |Bug ||Minor ||Resolved |Remove empty >>>>>>>>> column on the Logs view |1 |#2310 >>>>>>>>> |b0ba3c91044b2abd92289d9b3f7d913060ef6c31 >>>>>>>>> AIRFLOW-1226 |Bug ||Minor ||Resolved |Remove empty >>>>>>>>> column on the Jobs view |1 |#2309 >>>>>>>>> |c406652bd4c8f8099841c69f2d1ccc9678d77bfd >>>>>>>>> AIRFLOW-1221 |Bug ||Major ||Resolved |Fix >>>>>>>>> DatabricksSubmitRunOperator Templating |0 |- |- >>>>>>>>> AIRFLOW-1217 |Bug ||Major ||Resolved |Enable >>>>>> logging >>>>>>>>> in Sqoop hook |0 |- |- >>>>>>>>> AIRFLOW-1213 |Bug ||Major ||Resolved |Add hcatalog >>>>>>>>> parameters to the sqoop operator/hook|0 |- |- >>>>>>>>> AIRFLOW-1203 |Bug ||Major ||Closed |Tests >>> failing >>>>>>>>> after oauth upgrade |0 |- |- >>>>>>>>> AIRFLOW-1201 |Bug ||Major ||Resolved |Update >>>>>>>>> deprecated 'nose-parameterized' library to |0 |- |- >>>>>>>>> AIRFLOW-1200 |Bug ||Major ||Resolved |Forbid >>>>>>>>> creation of a variable with an empty key |1 |#2299 >>>>>>>>> |cac7d4cb571a4c1f7ba53360765ea6c640c719df >>>>>>>>> AIRFLOW-1199 |Bug ||Major ||Resolved |Create modal >>>>>>>>> looks like edit modal (edit tab selec|1 |#2293 >>>>>>>>> |51a83cce96d41eb1b9df9f82188c36c7e12061df >>>>>>>>> AIRFLOW-1197 |Bug ||Major ||Open |Contrib >>> Spark >>>>>>>>> Submit Hook on_kill error |0 |- |- >>>>>>>>> AIRFLOW-1192 |New Feature ||Major ||Resolved |Add support >>>>>> to >>>>>>>>> run Zeppelin Notebooks with Qubole |1 |#2322 >>>>>>>>> |bf966e63be2deb19fa2824efdfe854736318f1b5 >>>>>>>>> AIRFLOW-1191 |Improvement ||Major ||Open |Contrib >>> Spark >>>>>>>>> Submit hook should permit override o|0 |- |- >>>>>>>>> AIRFLOW-1186 |Bug ||Major ||Resolved |Task >>> duration >>>>>>>>> shows weird graph for some dags |1 |#2284 >>>>>>>>> |5f3ee52da7ca4f1fca8a84b3281b31aa6241514e >>>>>>>>> AIRFLOW-1180 |Bug ||Major ||Resolved |Fix >>> flask-wtf >>>>>>>>> version in setup.py for test_csrf_re|1 |#2280 >>>>>>>>> |93b8e9643cc273ec0ecc063b9f851d9a8012cd03 >>>>>>>>> AIRFLOW-1179 |Bug ||Major ||Closed |Pandas 0.20 >>>>>>>>> broke Google BigQuery hook |1 |#2279 >>>>>>>>> |c7c0b72d934fec10d890f7a3bd8fd06c0e2f44bd >>>>>>>>> AIRFLOW-1170 |Bug ||Major ||Resolved |DbApiHook >>>>>>>>> insert rows does not use prepared statem|1 |#2270 >>>>>>>>> |bb56805e3801c7f8d49acd8095fd3099609d6fe5 >>>>>>>>> AIRFLOW-1168 |Bug ||Major ||Resolved |DbApiHook >>>>>> does >>>>>>>>> not close connections when an excep|1 |#2269 >>>>>>>>> |777f181f81fa6a383a79b71fdcb9c4f4e77c785a >>>>>>>>> AIRFLOW-1167 |Bug ||Major ||Resolved |FTPHook file >>>>>>>>> modification date does not support op|1 |#2268 >>>>>>>>> |272c2f52142c167685646169431eb9dab85b7810 >>>>>>>>> AIRFLOW-1166 |Bug ||Major ||Resolved >>>>>>>>> |_change_state_for_tis_without_dagrun can take very|1 |#2267 >>>>>>>>> |2389a8a7e89ea949e8f9674a3d3b18d2142dfece >>>>>>>>> AIRFLOW-1160 |Bug ||Major ||Resolved |Upadte >>>>>>>>> SparkSubmitOperator parameters |1 |#2265 >>>>>>>>> |69bd26913824c5c796a0a828a5c43ab39d0902f8 >>>>>>>>> AIRFLOW-1150 |Bug ||Major ||Resolved |Fix scripts >>>>>>>>> execution in SparkSql hook |1 |#2259 >>>>>>>>> |093b2f04f7a010107fe6dfbc0658a53a1b8467e6 >>>>>>>>> AIRFLOW-1149 |Bug ||Major ||Resolved |Allow custom >>>>>>>>> filters to be added to jinja2 |1 |#2258 >>>>>>>>> |9692510c4e067d0e81fd8d5893bfb4a0e53941e2 >>>>>>>>> AIRFLOW-1145 |Bug ||Minor ||Resolved >>>>>>>>> |Closest_date_partition not working with before = T|1 |#2257 >>>>>>>>> |f446c0883cab30482436c47b867c4a440412f1ff >>>>>>>>> AIRFLOW-1136 |Bug ||Minor ||Resolved |Invalid >>>>>>>>> parameters are not captured for Sqoop oper|0 |- |- >>>>>>>>> AIRFLOW-1119 |Bug ||Major ||Resolved |Redshift to >>>>>> S3 >>>>>>>>> operator - headers not on first row|1 |#2245 >>>>>>>>> |6de5330d5336fb2455eea75c78e95f126320a5b0 >>>>>>>>> AIRFLOW-1089 |Improvement ||Major ||Resolved |Add Spark >>>>>>>>> application arguments to SparkSubmitOper|1 |#2229 >>>>>>>>> |b4e9eb8310957cac47f6964662ce16496061db2c >>>>>>>>> AIRFLOW-1081 |Bug ||Major ||Resolved |Task >>> duration >>>>>>>>> page is slow |0 |- |- >>>>>>>>> AIRFLOW-1078 |Bug ||Major ||Resolved |Latest_runs >>>>>>>>> endpoint broken in old flask versions |1 |#2224 >>>>>>>>> |a4083f367b1f0f6525bcd011122d854626ea6768 >>>>>>>>> AIRFLOW-1074 |Bug ||Major ||Open |Do not count >>>>>>>>> queued tasks in scheduler concurrency|1 |#2221 >>>>>>>>> |7a0284149a1dde4dc1d65a1d016eadf619e5320e >>>>>>>>> AIRFLOW-1064 |Improvement ||Major ||Resolved >>>>>>>>> |TaskInstanceModelView is slow |1 |#2215 >>>>>>>>> |a2c18a58d0ba14101ddb70403fb95feda73cf21a >>>>>>>>> AIRFLOW-1047 |Bug ||Major ||Resolved |Airflow logs >>>>>>>>> vulnerable to XSS |0 |- |- >>>>>>>>> AIRFLOW-1041 |Bug ||Major ||Resolved >>>>>> |DockerOperator >>>>>>>>> replaces its xcom_push method with |0 |- |- >>>>>>>>> AIRFLOW-1038 |Bug ||Major ||Resolved |Specify >>>>>> celery >>>>>>>>> serializers explicitly and pin vers|1 |#na >>>>>>>>> |d1c64ab7b4ee6215900a15849e890ed1b408f5f5 >>>>>>>>> AIRFLOW-1036 |Improvement ||Major ||Resolved |Exponential >>>>>>>>> backoff should use randomization |1 |#2262 >>>>>>>>> |b4ee88aa6d71145daf91fc71763950ede16f8383 >>>>>>>>> AIRFLOW-1007 |Bug ||Major ||Resolved |Jinja >>> sandbox >>>>>>>>> is vulnerable to RCE |0 |- |- >>>>>>>>> AIRFLOW-993 |Bug ||Minor ||Resolved |Dags should >>>>>>>>> modify the start date and end date of |1 |#2157 >>>>>>>>> |9fca409cebfd04ce8580c4657099a6db340eb7eb >>>>>>>>> AIRFLOW-945 |Bug ||Major ||Resolved |Revert >>>>>>>>> psycopg2 workaround when psycopg2 2.7.1 is |0 |- |- >>>>>>>>> AIRFLOW-935 |Bug ||Major ||Resolved |Impossible >>> to >>>>>>>>> use plugin executors |0 |- |- >>>>>>>>> AIRFLOW-860 |Bug ||Major ||Resolved |Circular >>>>>>>>> module dependency prevents loading of cus|0 |- |- >>>>>>>>> >>>>>>>>> >>>>>>>>> Max >>>>>>>>> >>>>>>>>> On Thu, Jun 8, 2017 at 12:18 AM, Bolke de Bruin <bdbr...@gmail.com >>>>>>>>> <mailto:bdbr...@gmail.com>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Max, >>>>>>>>>> >>>>>>>>>> Sounds good. Couple of things: >>>>>>>>>> >>>>>>>>>> * Can I suggest using the v1-8-test branch as the branch to be used >>>>>>> for >>>>>>>>>> preparing the rc? If we hit RC then move it over to v1-8-stable? >>>>>>>>> V1-8-test >>>>>>>>>> already had some fixes in that should land in 1.8.2 and the RC >>>>>> should >>>>>>>> be >>>>>>>>>> tagged in the stable branch. That also reduces to amount of merge >>>>>>>>> conflicts >>>>>>>>>> probably as many have been merged. Where did you branch off from? >>>>>>>>> Anyways, >>>>>>>>>> see also the release management thing on the wiki. >>>>>>>>>> >>>>>>>>>> Blocker(!) >>>>>>>>>> * In the backfills we can loose tasks to execute due to a task >>>>>>> setting >>>>>>>>> its >>>>>>>>>> own state to NONE if concurrency limits are reached, this makes >>>>>> them >>>>>>>> fall >>>>>>>>>> outside of the scope the backfill is managing hence they will not >>>>>> be >>>>>>>>>> executed (https://issues.apache.org/jira/browse/AIRFLOW-1294 >>>>>>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1294> < >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1294 >>>>>>>>>> <https://issues.apache.org/jira/browse/AIRFLOW-1294>>). Setting >>>>>> itself >>>>>>>> to >>>>>>>>>> NONE should probably be “CONCURRENCY_REACHED” (new state). I have >>>>>>>> marked >>>>>>>>> it >>>>>>>>>> as a blocker as we had multiple people hitting the issue, but I >>>>>> need >>>>>>>> 1-2 >>>>>>>>>> days to get a patch. Feel free to downgrade to critical if you like >>>>>>> :). >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> Bolke >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 8 Jun 2017, at 02:35, Maxime Beauchemin < >>>>>>>> maximebeauche...@gmail.com <mailto:maximebeauche...@gmail.com> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> What a pleasant, mind numbing afternoon doing some release >>>>>>> management >>>>>>>>>>> >>>>>>>>>>> Notes: >>>>>>>>>>> * Added a warning that the package name has changed on Pypi >>>>>>>>>>> <https://pypi.python.org/pypi/airflow >>>>>>>>>>> <https://pypi.python.org/pypi/airflow>> >>>>>>>>>>> * Removed references to my name here >>>>>>>>>>> <https://github.com/apache/incubator-airflow/pull/2352 >>>>>>>>>>> <https://github.com/apache/incubator-airflow/pull/2352>> and >>>>>> merged >>>>>>>>>>> * Addressed John D. Ament's concerns here >>>>>>>>>>> <https://github.com/apache/incubator-airflow/pull/2354 >>>>>>>>>>> <https://github.com/apache/incubator-airflow/pull/2354>>, please >>>>>>>>> review! >>>>>>>>>>> * "footable" appears to have been removed, not a problem anymore >>>>>>>>>>> * that `airflow-jira is a god send! thanks Bolke. >>>>>>>>>>> * reviewed list of Airbnb's production cherries and flagged those >>>>>>> as >>>>>>>>> `Fix >>>>>>>>>>> Version == 1.8.2` >>>>>>>>>>> * Started branch v1-8-2.rc1 and started picking cherries using >>>>>>>>>>> `airflow-jira compare 1.8.2` >>>>>>>>>>> >>>>>>>>>>> I'll finish going through picking everything that targeted 1.8.2 >>>>>>> that >>>>>>>>>> does >>>>>>>>>>> not create merge conflict. >>>>>>>>>>> >>>>>>>>>>> If there's anything flagged as "blocker" that generates merge >>>>>>>> conflict, >>>>>>>>>>> I'll go case by case about it. >>>>>>>>>>> >>>>>>>>>>> Soon after, I should be able to announce 1.8.2 RC1, hopefully >>>>>>>> sometime >>>>>>>>>>> tomorrow or Friday. >>>>>>>>>>> >>>>>>>>>>> Let me know if there's anything else I'm missing that I should >>>>>>>>> consider. >>>>>>>>>>> >>>>>>>>>>> Cheers! >>>>>>>>>>> >>>>>>>>>>> Max >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >>> >
