Probably as a result of a cherry pick and some migrations not being aplied to the v1-10-test branch.
I feel this is something we can fix up in on master branch/for 1.10.7 to bring them back in line. (So long as `airflow upgradedb` works then I'm happy.) -ash > On 27 Oct 2019, at 16:12, Driesprong, Fokko <fo...@driesprong.frl> wrote: > > Yes, that's true. But I expect the 1.x to be a subset of the 2.0 > migrations, so somewhere it went south. > > Right now we have 3 heads in 1.10.6rc2: > MacBook-Pro-van-Fokko:airflow fokkodriesprong$ git checkout 1.10.6rc2 > MacBook-Pro-van-Fokko:airflow fokkodriesprong$ alembic heads > a56c9515abdc (head) > 004c1210f153 (head) > 74effc47d867 (head) > > Full history: > > c8ffec048a3b -> a56c9515abdc (head), Remove dag_stat table > 939bb1e647c8 -> 004c1210f153 (head), increase queue name size limit > 6e96a59344a4 -> 74effc47d867 (head), change datetime to datetime2(6) on > MSSQL tables > 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable > dd4ecb8fbee3 -> 939bb1e647c8 (branchpoint), task reschedule fk on cascade > delete > c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag > 41f5f12752f8 -> c8ffec048a3b (branchpoint), add fields to dag > 03bc53e68815 -> 41f5f12752f8, add superuser field > 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815 (mergepoint), merge_heads_2 > 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table > dd25f486b8ea -> bf00311e1990, add index to taskinstance > 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag > 856955da8476 -> 9635ae0956e7 (branchpoint), index-faskfail > f23433877c24 -> 856955da8476, fix sqlite foreign key > 05f30312d566 -> f23433877c24, fix mysql not null constraint > 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566 (mergepoint), merge heads > 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness > 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing > d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing > d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness > 947454bf1dff -> d2ae31099d61 (branchpoint), Increase text size for MySQL > (not relevant for other DBs' text types) > bdaa763e6c56 -> 947454bf1dff, add ti job_id index > cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary > 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance > 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table > 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance > 4addfa1236f1 -> 8504051e801b, xcom dag task indices > f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables > 64de9cddf6c9 -> f2ca10b85618, add dag_stats table > 211e584da130 -> 64de9cddf6c9, add task fails journal table > 2e82aab8ef20 -> 211e584da130, add TI state index > 1968acfc09e3 -> 2e82aab8ef20, rename user table > bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table > bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of > the 'Extra' field in connection > 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss > 561833c1c74b -> 4446e08588, dagrun start end > 40e67319e3a9 -> 561833c1c74b, add password column to user > 2e541a1dcfed -> 40e67319e3a9, dagrun_config > 1b38cef5b76e -> 2e541a1dcfed, task_duration > 502898887f84 -> 1b38cef5b76e, add dagrun > 52d714495f0 -> 502898887f84, Adding extra to Log > 338e90f54d61 -> 52d714495f0, job_id indices > 13eb55f81627 -> 338e90f54d61, More logging into task_instance > 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with > earlier migrations > e3a246e0dc1 -> 1507a7289a2f, create is_encrypted > <base> -> e3a246e0dc1, current schema > > Cheers, Fokko > > Op zo 27 okt. 2019 om 16:48 schreef Kaxil Naik <kaxiln...@gmail.com>: > >> As 2.0 is not released, if migration are broken, we can still fix them on >> the mater itself. >> >> >> >> On Sun, Oct 27, 2019, 15:29 Driesprong, Fokko <fo...@driesprong.frl> >> wrote: >> >>> The problem is that if you upgrade from the 1.10 branch to 2.0, the >>> migrations don't work. To probably the migrations in 1.10 and 2.0 have >>> diverged somewhere. >>> >>> The following can be used to reproduce this: >>> git pull upstream master # ensure we have latest >>> git checkout v-10-stable # 1.10.6rc2 >>> rm ~/airflow/airflow.db >>> airflow initdb >>> git checkout master >>> airflow db upgrade >>> >>> >>> >>> Op zo 27 okt. 2019 om 15:53 schreef Ash Berlin-Taylor <a...@apache.org>: >>> >>>> That PR you mentioned isn't yet in a release, so can we not just edit >>> that >>>> migration in place? >>>> >>>> I'm not quite following (I didn't get much sleep) but I don't see how >>>> broken code on master affects this release? >>>> >>>> -a >>>> >>>>> On 27 Oct 2019, at 14:28, Driesprong, Fokko <fo...@driesprong.frl> >>>> wrote: >>>>> >>>>> I'm having second thoughts on my +1 vote. >>>>> >>>>> It turns out that the database migrations are broken from 1.10.6 to >>>> 2.0.0: >>>>> >>>> >>> >> https://github.com/apache/airflow/pull/6370/files/c9b9312a60f891475d3072584171c2af56246829#r339287344 >>>>> >>>>> So we either need to release a 1.10.7 and force users to migrate to >>> that >>>>> version first, before going to 2.0.0, otherwise, it won't work. I've >>>> opened >>>>> up a PR to fix this for 2.0.0: >>>> https://github.com/apache/airflow/pull/6442 >>>>> >>>>> Cheers, Fokko >>>>> >>>>> >>>>> Op za 26 okt. 2019 om 09:23 schreef Jarek Potiuk < >>>> jarek.pot...@polidea.com>: >>>>> >>>>>> +1 (binding) >>>>>> >>>>>> Tested on Python 3.7/Python 2.7 using local installation + double >>>> checked >>>>>> sources with RAT licence tool (All good this time!). >>>>>> >>>>>> Regarding why AIRFLOW-5746 ( >>> https://github.com/apache/airflow/pull/6434 >>>> ) >>>>>> is >>>>>> being reverted (or rather being improved on). I don't think it's a >>>> blocker >>>>>> for this release. >>>>>> >>>>>> It is just a test change - the tests how they were written, broke >> some >>>>>> development environments (the airflow resetdb was failing in case >>>>>> PYTHONPATH was not specially crafted). The current test does not >>> really >>>>>> test what it should (passing PYTHONPATH to impersonated tasks) but >> the >>>>>> functionality (impersonation + path) has not actually changed and it >>>> works. >>>>>> The test worked fine before the change was added - it's just imports >>> in >>>>>> tests that have been changed. >>>>>> >>>>>> Kamil (thanks!) already submitted a fix for that, that solves it >>>>>> permanently https://github.com/apache/airflow/pull/6436/ - will >>> review >>>> and >>>>>> merge soon. But it's not a blocker IMHO. >>>>>> >>>>>> J. >>>>>> >>>>>> On Sat, Oct 26, 2019 at 2:10 AM Kaxil Naik <kaxiln...@gmail.com> >>> wrote: >>>>>> >>>>>>> +1 (binding) >>>>>>> >>>>>>> On Sat, Oct 26, 2019 at 12:05 AM Driesprong, Fokko >>>> <fo...@driesprong.frl >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 binding from my side >>>>>>>> >>>>>>>> Ran an example DAGs with Docker using Python 3.7. >>>>>>>> >>>>>>>> We might need to check why AIRFLOW-5746 is being reverted: >>>>>>>> https://github.com/apache/airflow/pull/6434 >>>>>>>> >>>>>>>> If there is another RC, I'd like to request to cherry-pick >>>>>>>> https://github.com/apache/airflow/pull/6370 onto the 1.10 branch. >>>>>>>> >>>>>>>> Cheers, Fokko >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Op vr 25 okt. 2019 om 23:04 schreef Ash Berlin-Taylor < >>> a...@apache.org >>>>>>> : >>>>>>>> >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I have cut Airflow 1.10.6 RC2. This email is calling a vote on >> the >>>>>>>>> release, which will last for 72 hours, until Monday 28, October >>> 22nd >>>>>> at >>>>>>>>> 21:05 UTC. >>>>>>>>> >>>>>>>>> Consider this my (binding) +1. >>>>>>>>> >>>>>>>>> Airflow 1.10.6 RC1 is available at: < >>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/1.10.6rc2/> >>>>>>>>> >>>>>>>>> *apache-airflow-1.10.6rc2-source.tar.gz* is a source release that >>>>>> comes >>>>>>>>> with INSTALL instructions. >>>>>>>>> *apache-airflow-1.10.6rc2-bin.tar.gz* is the binary Python >> "sdist" >>>>>>>> release. >>>>>>>>> *apache_airflow-1.10.6rc2-py2.py3-none-any.whl* is the binary >>> Python >>>>>>>>> "wheel" release. >>>>>>>>> >>>>>>>>> Public keys are available at: < >>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS> >>>>>>>>> >>>>>>>>> As per normal the rc1 is available for testing from PyPi. >>>>>>>>> >>>>>>>>> Only votes from PMC members are binding, but members of the >>> community >>>>>>> are >>>>>>>>> encouraged to test the release and vote with "(non-binding)". >>>>>>>>> >>>>>>>>> Please note that the version number excludes the `rcX` string, so >>>>>> it's >>>>>>>> now >>>>>>>>> simply 1.10.6. This will allow us to rename the artifact without >>>>>>>> modifying >>>>>>>>> the artifact checksums when we actually release. >>>>>>>>> >>>>>>>>> The changes since RC1 are to fix License issues, ensure tests are >>>>>>> running >>>>>>>>> on Py2 (they weren't, but the only py3 bits that crept in were in >>> the >>>>>>>> test >>>>>>>>> files luckily.). >>>>>>>>> >>>>>>>>> Changelog since 1.10.6rc1: >>>>>>>>> >>>>>>>>> * 73bf71835 [AIRFLOW-XXX] Update date in changelog [Ash >>>>>> Berlin-Taylor] >>>>>>>>> * 143b43151 [AIRFLOW-5750] Licence check is done also for >>>>>>> non-executable >>>>>>>>> .sh (#6425) [Jarek Potiuk] >>>>>>>>> * 544f2b336 [AIRFLOW-5754] Improved RAT checking (#6429) [Jarek >>>>>> Potiuk] >>>>>>>>> * 7904669ca [AIRFLOW-5755] Fixed most problems with py27 [Jarek >>>>>> Potiuk] >>>>>>>>> * d601752c4 [AIRFLOW-5748] Remove python auto-detection (#6423) >>>>>> [Jarek >>>>>>>>> Potiuk] >>>>>>>>> * 71e20417f [AIRFLOW-5746] Fix problems with static checks >> (#6420) >>>>>>> [Jarek >>>>>>>>> Potiuk] >>>>>>>>> * 7a6adad60 [AIRFLOW-5746] move FakeDateTime into the only place >> it >>>>>> is >>>>>>>>> used (#6416) [Michael R. Crusoe] >>>>>>>>> * e30fb85ca [AIRFLOW-5745] Breeze complete has now licence >> (#6415) >>>>>>> [Jarek >>>>>>>>> Potiuk] >>>>>>>>> >>>>>>>>> Files changes since rc1: >>>>>>>>> >>>>>>>>> airflow ❯ git diff --stat 1.10.6rc1...1.10.6rc2 >>>>>>>>> .pre-commit-config.yaml | 1 - >>>>>>>>> .rat-excludes | 1 + >>>>>>>>> .travis.yml | 47 >>>>>>>>> +++++++++++++++++++++++++++++++++++++++-------- >>>>>>>>> CHANGELOG.txt | 2 +- >>>>>>>>> Dockerfile-checklicence | 2 +- >>>>>>>>> airflow/models/dag.py | 5 ++++- >>>>>>>>> breeze | 1 + >>>>>>>>> breeze-complete | 17 >> +++++++++++++++++ >>>>>>>>> common/_autodetect_variables.sh | 49 >>>>>>>>> +++++++++++++------------------------------------ >>>>>>>>> files/x | 0 >>>>>>>>> files/y | 0 >>>>>>>>> scripts/ci/_utils.sh | 16 >> ++++++++++++++-- >>>>>>>>> scripts/ci/ci_check_license.sh | 1 + >>>>>>>>> scripts/ci/ci_run_airflow_testing.sh | 2 +- >>>>>>>>> scripts/ci/in_container/run_check_licence.sh | 2 +- >>>>>>>>> tests/contrib/hooks/test_ssh_hook.py | 3 ++- >>>>>>>>> tests/dags/test_impersonation_custom.py | 12 +++++++++++- >>>>>>>>> tests/models/test_baseoperator.py | 9 ++++----- >>>>>>>>> tests/models/test_dag.py | 4 ++-- >>>>>>>>> tests/test_sentry.py | 2 +- >>>>>>>>> tests/test_utils/fake_datetime.py | 29 >>>>>>>>> ----------------------------- >>>>>>>>> tests/test_utils/system_tests_class.py | 6 +++++- >>>>>>>>> tests/www/test_views.py | 6 +++--- >>>>>>>>> 23 files changed, 122 insertions(+), 95 deletions(-) >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>> >>>> >>> >>
