See this PR: https://github.com/apache/airflow/pull/21378 (Particularly
last commit).
Currently failing on pyodbc.OperationalError: ('08001', '[08001]
[Microsoft][ODBC Driver 18 for SQL Server]SSL Provider: [error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify failed:self
signed certificate] (-1) (SQLDriverConnect)' but that will fixed in the nex
fixup.
On Wed, Feb 16, 2022 at 8:45 PM Alex Begg <[email protected]> wrote:
> Not sure if this is done already, but I think places with "ODBC Driver 17
> for SQL Server" hardcoded in might need to change from 17 to 18 as well:
> https://github.com/apache/airflow/search?q=ODBC+Driver+17+for+SQL+Server
>
>
> On Wed, Feb 16, 2022 at 12:06 AM Jarek Potiuk <[email protected]> wrote:
>
>> Yep. Already did :)
>>
>> On Wed, Feb 16, 2022 at 6:22 AM Alex Begg <[email protected]> wrote:
>>
>>> I wanted to point out that not only did Microsoft add Debian Bullseye
>>> support for MSSQL today, to do so they did a major version
>>> upgrade, from msodbcsql17 to msodbcsql18. So in addition to updating the
>>> URL used to populate mssql-release.list, you also need to change to
>>> msodbcsql18. I see msodbcsql17 is in multiple places in the Airflow code
>>> base.
>>>
>>> Instructions for installing msodbcsql18 in Debian are here:
>>>
>>> https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15#debian18
>>>
>>> Alex
>>>
>>>
>>> On Tue, Feb 15, 2022 at 1:32 AM Jarek Potiuk <[email protected]> wrote:
>>>
>>>> Yeah. That would definitely be a simpler approach for us.
>>>>
>>>> What we can also do is we can still make sure that you can actually
>>>> "build" Buster images if you want and we could run the simple tests we
>>>> already have for those images (those tests just check very basic stuff
>>>> - like if expected packages are importable, if Airflow commands work,
>>>> if you can run airflow as root user etc..
>>>> This would be very little to no complication of the CI and we could
>>>> provide a recipe for the users how to build their own buster image -
>>>> but without guarantees that everything works there.
>>>>
>>>> This way might provide the users with a "workable solution" while we
>>>> could keep it simple for us and put a strong emphasis on "If you want
>>>> to get full support - also upgrade to the latest stable OS".
>>>>
>>>> J.
>>>>
>>>>
>>>> On Tue, Feb 15, 2022 at 9:53 AM Ash Berlin-Taylor <[email protected]>
>>>> wrote:
>>>> >
>>>> > My vote is to just move forward and only release Bullseye images for
>>>> 2.3 onwards
>>>> >
>>>> > On 15 February 2022 08:43:50 GMT, Jarek Potiuk <[email protected]>
>>>> wrote:
>>>> >>
>>>> >> Good news from today - I got the response from Microsoft and it seems
>>>> >> that MSSQL Bullseye support is going to be released today :) . So at
>>>> >> least that problem is going to be solved. Still the question of
>>>> >> whether we should build/release both Bullseye and Buster ?
>>>> >>
>>>> >> J.
>>>> >>
>>>> >> On Mon, Feb 14, 2022 at 2:14 PM Jarek Potiuk <[email protected]>
>>>> wrote:
>>>> >>>
>>>> >>>
>>>> >>> And Python default images are Bullseye now.
>>>> >>>
>>>> >>> On Mon, Feb 14, 2022 at 2:13 PM Jarek Potiuk <[email protected]>
>>>> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>> Buster end of life is August 2022 of course :)
>>>> >>>>
>>>> >>>> On Mon, Feb 14, 2022 at 2:12 PM Jarek Potiuk <[email protected]>
>>>> wrote:
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Hello everyone,
>>>> >>>>>
>>>> >>>>> I am just about to complete preparation to make our images (and
>>>> CI) work on Debian 11 images (Bullseye) vs Debian 10 (Buster). All looks
>>>> (almost) good, but I would like to tap into the collective wisdom of the
>>>> group here to decide on next steps (and maybe propose a policy we might be
>>>> following in the future). The last change that allows to easily switch
>>>> between the two is in reviee:
>>>> https://github.com/apache/airflow/pull/21546
>>>> >>>>>
>>>> >>>>> Context/where we are:
>>>> >>>>>
>>>> >>>>> For quite a while we used Buster (Debian 10) as our base image.
>>>> Buster is a stable release and it is being replaced with the next stable
>>>> release Bullseye (Debian 11).
>>>> >>>>>
>>>> >>>>> The Release schedule is here:
>>>> https://wiki.debian.org/DebianReleases
>>>> >>>>>
>>>> >>>>> The most important facts from those:
>>>> >>>>> * Bullseye was introduced in August 2021 (6 months ago) and has
>>>> no end-of-life yet
>>>> >>>>> * Buster was introduced in July 2019 and its end of life is ~
>>>> August 2020 (approx. 6 months from now)
>>>> >>>>>
>>>> >>>>> Python default images are (as of a few months), However they
>>>> provide (as usual -buster and -bullseye) specific images.
>>>> >>>>>
>>>> >>>>> The Bullseye switch is really needed to support ARM (M1) images
>>>> because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989604
>>>> (still unresolved on Buster).
>>>> >>>>>
>>>> >>>>> It seems appropriate to switch to Bullseye now, however there is
>>>> one problem with it - MSSQL drivers do not yet support Bullseye. I've
>>>> reached out to the maintainers and await their answer when it might be
>>>> available:
>>>> https://github.com/MicrosoftDocs/sql-docs/issues/7255#issuecomment-1037097131
>>>> . This is the last blocker that prevents full switch (failing PR for a full
>>>> switch is https://github.com/apache/airflow/pull/21378). All tests
>>>> pass except MSSQL.
>>>> >>>>>
>>>> >>>>> But I think we need to proceed, regardless of the timeline and
>>>> switch to Bullseye.
>>>> >>>>>
>>>> >>>>> Question:
>>>> >>>>>
>>>> >>>>> We need to decide on what support we give our users in the
>>>> images now, in the 2.3.0 release and in the future. We have no policies for
>>>> that yet, so it might be a good time to decide now. Some of the users might
>>>> "rely" on the Buster being used by the images as they might have some
>>>> incompatible libraries (similarly to MSSQL). And I believe we need to
>>>> support Buster till at least the end of life of it.
>>>> >>>>>
>>>> >>>>> This support might be twofold:
>>>> >>>>>
>>>> >>>>> * we can publish in https://hub.docker.com/r/apache/airflow our
>>>> binary images with Buster and/or Bullseye
>>>> >>>>> * we can only publish one of those (and change from Buster to
>>>> Bullseye for 2.3.0) but users will be able to build their own custom images
>>>> using our Dockerfile for either of those.
>>>> >>>>>
>>>> >>>>> Proposal:
>>>> >>>>>
>>>> >>>>> I think there are multiple reasons why we should support both
>>>> Bullseye and Buster for the coming 6 months. We might build both images in
>>>> CI and add a new "matrix" dimension and run most tests on Bullseye but some
>>>> tests on Buster (specifically MSSQL until Bullseye is supported).
>>>> >>>>>
>>>> >>>>> We can publish both types of images (with -buster, -bullseye
>>>> suffixes same as Python) but the default image should be changed to
>>>> Bullseye.
>>>> >>>>>
>>>> >>>>> This will add some complexity (and build overhead) to our CI,
>>>> but I think it's worth it.
>>>> >>>>>
>>>> >>>>> Then we could drop support and release the -buster images after
>>>> it reaches end of life (but we will leave the users possibility of building
>>>> their own buster images without guarantee that it will work).
>>>> >>>>>
>>>> >>>>> I think this might become our Policy also for the future (6
>>>> months before end-of-life we switch by default to the new stable and
>>>> provide built images until end-of-life).
>>>> >>>>>
>>>> >>>>> WDYT? Does it look like a good proposal?
>>>> >>>>>
>>>> >>>>> J.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>>
>>>
