setting up software can be tricky because some people are installing on clean machines while others already have the required libraries installed. Some people go straight to docker :)
On this page https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database it only says "We recommend using the mysqlclient driver and specifying it in your SqlAlchemy connection string." which I found vague. I think the page should mention running the *airflow* command $ airflow providers list # to check and see if apache-airflow-providers-mysql is listed if not install then install by running the command $ pip install mysqlclient then check again to see if airflow can see sqlclient. $ airflow providers list after that specify mysqlient in your SqlAlchemy connection string by setting sql_alchemy_conn = mysql+mysqldb://<user>:<password>@<host>[:<port>]/<dbname> in the airflow.cfg This is a history of my commands, it shows the software I ran and installed. 657 airflow providers list 704 sudo apt-get install python3-dev default-libmysqlclient-dev build-essential pkg-config 705 pip install mysqlclient 706 airflow providers list 707 sudo apt install python3-devel mysql-devel 709 pip install mysql-devel 710 airflow standalone 711 pip install apache-airflow-providers-mysql 712 airflow standalone 730 airflow providers list 704 sudo apt-get install python3-dev default-libmysqlclient-dev build-essential pkg-config 705 pip install mysqlclient 706 airflow providers list 707 sudo apt install python3-devel mysql-devel 709 pip install mysql-devel 710 airflow standalone 711 pip install apache-airflow-providers-mysql 712 airflow standalone 730 airflow providers list http://www.frontbutton.com/ ¯\_(ツ)_/¯ ♡۶♡ ۶♡۶ <http://www.backbutton.co.uk> On Sat, 23 Dec 2023 at 01:53, Jarek Potiuk <ja...@potiuk.com> wrote: > Sure - anything more you think would be useful is a great contribution to > the docs. > > On Fri, Dec 22, 2023 at 9:48 PM Zahid Rahman <zahidr1...@gmail.com> wrote: > >> >> Can we count on this ? >> Sure I looked all over the internet to configure mysql to overcome the >> warning message. >> There is a lot more work required then I have mentioned so far. >> >> http://www.frontbutton.com/ >> ¯\_(ツ)_/¯ >> ♡۶♡ ۶♡۶ >> >> <http://www.backbutton.co.uk> >> >> >> On Sat, 23 Dec 2023 at 00:28, Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> > *What about* variable sql_engine_collation_for_ids=utf8mb3_bin >>> mentioned next on same page >>> > >>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>> > should I set that in /etc/mysql/my.cnf also ? >>> >>> > In the airflow.cfg the default setting is sql_engine_encoding = utf-8 >>> > Should I change from sql_engine_encoding = utf-8 in airflow,cfg to >>> sql_engine_encoding = utf8mb4 in airflow.cfg if I am changing >>> sql_engine_collation_for_ids=utf8mb3_bin in /etc/mysql/my.cnf * ?* >>> >>> No - those are "airflow" configuration settings, not MySQL driver >>> settings. The former are described in >>> https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html, >>> the latter - in the driver documentation (when you pass the URL to >>> sqlalchemy, you actually select the driver to use, and this driver >>> parameters (`mysql+mysqldb`) has its own parameters - those are configured >>> in a way that is specific for the driver (so my.cnf in your case). For >>> example in case of mysql +mysqldb - you can start from this page - which >>> describes mysql dialects of sqlalchemy >>> https://docs.sqlalchemy.org/en/20/dialects/mysql.html >>> >>> >>> > Perhaps you may wish to update the page. >>> >>> Absolutely Zahid. And I have a kind request after you got all the >>> explanations - you are absolutely welcome to propose a PR. Can I ask you to >>> help with it please? >>> >>> When you look at the page with documentation - you can see "Suggest a >>> change on this page" at the bottom right. If you click it, a PR will be >>> opened for you and you will be able to modify the sources of the page in >>> the way you think is best for people like you to better understand. You are >>> probably one of the best people now to update the docs as you struggled >>> with it and got confused and then got helped. So giving back to the >>> community (like 2700+ other contributors) by making the documentation >>> clearer is a great idea. >>> >>> It would be fantastic if a person like you phrases it in a way that will >>> be easier to digest for people like you. >>> >>> Can we count on this ? >>> >>> J. >>> >>> On Fri, Dec 22, 2023 at 5:36 PM Zahid Rahman <zahidr1...@gmail.com> >>> wrote: >>> >>>> >>>> Thanks, I have updated my.cnf >>>> $ cat /etc/mysql/my.cnf >>>> # >>>> # The MySQL database server configuration file. >>>> # >>>> # You can copy this to one of: >>>> # - "/etc/mysql/my.cnf" to set global options, >>>> # - "~/.my.cnf" to set user-specific options. >>>> # >>>> # One can use all long options that the program supports. >>>> # Run program with --help to get a list of available options and with >>>> # --print-defaults to see which it would actually understand and use. >>>> # >>>> # For explanations see >>>> # http://dev.mysql.com/doc/mysql/en/server-system-variables.html >>>> >>>> # >>>> # * IMPORTANT: Additional settings that can override those from this >>>> file! >>>> # The files must end with '.cnf', otherwise they'll be ignored. >>>> # >>>> >>>> !includedir /etc/mysql/conf.d/ >>>> !includedir /etc/mysql/mysql.conf.d/ >>>> explicit_defaults_for_timestamp=1 >>>> --------------------------EOF my.cnf -------------------- >>>> >>>> When I said it worked I meant the two warning messages disappeared when >>>> you first start >>>> airflow with commands >>>> $ airflow standalone >>>> by making these two changes in airflow.cfg >>>> *from * >>>> executor = SequentialExecutor >>>> >>>> *to*executor = LocalExecutor >>>> *&* >>>> *from* >>>> sql_alchemy_conn = sqlite:////home/zahid/airflow/airflow.db >>>> *to* >>>> sql_alchemy_conn = mysql+mysqldb://airflow_user:airflow_pass@localhost >>>> :3306/airflow_db >>>> >>>> >>>> *What about* variable sql_engine_collation_for_ids=utf8mb3_bin >>>> mentioned next on same page >>>> >>>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>>> should I set that in /etc/mysql/my.cnf also ? >>>> >>>> In the airflow.cfg the default setting is sql_engine_encoding = utf-8 >>>> Should I change from sql_engine_encoding = utf-8 in airflow,cfg to >>>> sql_engine_encoding = utf8mb4 in airflow.cfg if I am changing >>>> sql_engine_collation_for_ids=utf8mb3_bin in /etc/mysql/my.cnf * ?* >>>> >>>> >>>> Regards >>>> Zahid >>>> >>>> http://www.frontbutton.com/ >>>> ¯\_(ツ)_/¯ >>>> ♡۶♡ ۶♡۶ >>>> >>>> <http://www.backbutton.co.uk> >>>> >>>> >>>> On Fri, 22 Dec 2023 at 19:33, Lars Winderling < >>>> lars.winderl...@posteo.de> wrote: >>>> >>>>> Hi Zahid, >>>>> >>>>> the additional MySQL config for the timestamps should go into a config >>>>> file for MySQL. By default, it is called my.cnf. But you can pass it as a >>>>> cli option, when starting the mysql daemon (as cited in your message). As >>>>> an option to the mysql daemon mysqld, it defines the behavior of only the >>>>> mysql daemon, so you cannot set it on client side. The airflow.cfg is the >>>>> wrong place. It might work, but maybe in the long run not as expected due >>>>> to quirks with MySQL not following the SQL standard very closely. >>>>> Best, Lars >>>>> >>>>> >>>>> On 22 December 2023 15:07:07 CET, Zahid Rahman <zahidr1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> It was unclear from this page >>>>>> >>>>>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>>>>> which is a link from the airflow home when airflow first started. >>>>>> that I need to update the *airflow.cfg* to configure the mysql >>>>>> database like so. >>>>>> >>>>>> # sql_alchemy_conn = sqlite:////home/zahid/airflow/airflow.db >>>>>> sql_alchemy_conn = mysql+mysqldb://airflow_user:airflow_pass@localhost >>>>>> :3306/airflow_db >>>>>> >>>>>> Perhaps you may wish to update the page. >>>>>> >>>>>> ALSO >>>>>> I did not make any change to file "my.cnf" nor do I have one that I >>>>>> found in the ~/airflow as described below. >>>>>> >>>>>> "We rely on more strict ANSI SQL settings for MySQL in order to have >>>>>> sane defaults. Make sure to have specified >>>>>> explicit_defaults_for_timestamp=1 option under [mysqld] section in >>>>>> your my.cnf file. You can also activate these options with the >>>>>> --explicit-defaults-for-timestamp switch passed to mysqld executable" >>>>>> >>>>>> This instruction did confuse me a bit at first until I took the >>>>>> guess to make change in the airflow.cfg and *it worked!* >>>>>> >>>>>> Regards >>>>>> Zahid >>>>>> >>>>>> http://www.frontbutton.com/ >>>>>> ¯\_(ツ)_/¯ >>>>>> ♡۶♡ ۶♡۶ >>>>>> >>>>>> <http://www.backbutton.co.uk> >>>>>> >>>>>
