Again - cool. Feel free to make PR from those proposals by clicking "Suggest a change on this page".
On Fri, Dec 22, 2023 at 11:00 PM Zahid Rahman <zahidr1...@gmail.com> wrote: > setting up software can be tricky because some people are installing on > clean machines while others already have the required libraries installed. > Some people go straight to docker :) > > On this page > https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database > it only says "We recommend using the mysqlclient driver and specifying > it in your SqlAlchemy connection string." which I found vague. > > I think the page should mention running the *airflow* command > $ airflow providers list # to check and see if > apache-airflow-providers-mysql is listed > if not install then install by running the command > $ pip install mysqlclient > then check again to see if airflow can see sqlclient. > $ airflow providers list > after that specify mysqlient in your SqlAlchemy connection string by > setting > sql_alchemy_conn = > mysql+mysqldb://<user>:<password>@<host>[:<port>]/<dbname> > in the airflow.cfg > > This is a history of my commands, it shows the software I ran and > installed. > > 657 airflow providers list > 704 sudo apt-get install python3-dev default-libmysqlclient-dev > build-essential pkg-config > 705 pip install mysqlclient > 706 airflow providers list > 707 sudo apt install python3-devel mysql-devel > 709 pip install mysql-devel > 710 airflow standalone > 711 pip install apache-airflow-providers-mysql > 712 airflow standalone > 730 airflow providers list > 704 sudo apt-get install python3-dev default-libmysqlclient-dev > build-essential pkg-config > 705 pip install mysqlclient > 706 airflow providers list > 707 sudo apt install python3-devel mysql-devel > 709 pip install mysql-devel > 710 airflow standalone > 711 pip install apache-airflow-providers-mysql > 712 airflow standalone > 730 airflow providers list > > http://www.frontbutton.com/ > ¯\_(ツ)_/¯ > ♡۶♡ ۶♡۶ > > <http://www.backbutton.co.uk> > > > On Sat, 23 Dec 2023 at 01:53, Jarek Potiuk <ja...@potiuk.com> wrote: > >> Sure - anything more you think would be useful is a great contribution to >> the docs. >> >> On Fri, Dec 22, 2023 at 9:48 PM Zahid Rahman <zahidr1...@gmail.com> >> wrote: >> >>> >> Can we count on this ? >>> Sure I looked all over the internet to configure mysql to overcome the >>> warning message. >>> There is a lot more work required then I have mentioned so far. >>> >>> http://www.frontbutton.com/ >>> ¯\_(ツ)_/¯ >>> ♡۶♡ ۶♡۶ >>> >>> <http://www.backbutton.co.uk> >>> >>> >>> On Sat, 23 Dec 2023 at 00:28, Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> > *What about* variable sql_engine_collation_for_ids=utf8mb3_bin >>>> mentioned next on same page >>>> > >>>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>>> > should I set that in /etc/mysql/my.cnf also ? >>>> >>>> > In the airflow.cfg the default setting is sql_engine_encoding = utf-8 >>>> > Should I change from sql_engine_encoding = utf-8 in airflow,cfg to >>>> sql_engine_encoding = utf8mb4 in airflow.cfg if I am changing >>>> sql_engine_collation_for_ids=utf8mb3_bin in /etc/mysql/my.cnf * ?* >>>> >>>> No - those are "airflow" configuration settings, not MySQL driver >>>> settings. The former are described in >>>> https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html, >>>> the latter - in the driver documentation (when you pass the URL to >>>> sqlalchemy, you actually select the driver to use, and this driver >>>> parameters (`mysql+mysqldb`) has its own parameters - those are configured >>>> in a way that is specific for the driver (so my.cnf in your case). For >>>> example in case of mysql +mysqldb - you can start from this page - which >>>> describes mysql dialects of sqlalchemy >>>> https://docs.sqlalchemy.org/en/20/dialects/mysql.html >>>> >>>> >>>> > Perhaps you may wish to update the page. >>>> >>>> Absolutely Zahid. And I have a kind request after you got all the >>>> explanations - you are absolutely welcome to propose a PR. Can I ask you to >>>> help with it please? >>>> >>>> When you look at the page with documentation - you can see "Suggest a >>>> change on this page" at the bottom right. If you click it, a PR will be >>>> opened for you and you will be able to modify the sources of the page in >>>> the way you think is best for people like you to better understand. You are >>>> probably one of the best people now to update the docs as you struggled >>>> with it and got confused and then got helped. So giving back to the >>>> community (like 2700+ other contributors) by making the documentation >>>> clearer is a great idea. >>>> >>>> It would be fantastic if a person like you phrases it in a way that >>>> will be easier to digest for people like you. >>>> >>>> Can we count on this ? >>>> >>>> J. >>>> >>>> On Fri, Dec 22, 2023 at 5:36 PM Zahid Rahman <zahidr1...@gmail.com> >>>> wrote: >>>> >>>>> >>>>> Thanks, I have updated my.cnf >>>>> $ cat /etc/mysql/my.cnf >>>>> # >>>>> # The MySQL database server configuration file. >>>>> # >>>>> # You can copy this to one of: >>>>> # - "/etc/mysql/my.cnf" to set global options, >>>>> # - "~/.my.cnf" to set user-specific options. >>>>> # >>>>> # One can use all long options that the program supports. >>>>> # Run program with --help to get a list of available options and with >>>>> # --print-defaults to see which it would actually understand and use. >>>>> # >>>>> # For explanations see >>>>> # http://dev.mysql.com/doc/mysql/en/server-system-variables.html >>>>> >>>>> # >>>>> # * IMPORTANT: Additional settings that can override those from this >>>>> file! >>>>> # The files must end with '.cnf', otherwise they'll be ignored. >>>>> # >>>>> >>>>> !includedir /etc/mysql/conf.d/ >>>>> !includedir /etc/mysql/mysql.conf.d/ >>>>> explicit_defaults_for_timestamp=1 >>>>> --------------------------EOF my.cnf -------------------- >>>>> >>>>> When I said it worked I meant the two warning messages disappeared >>>>> when you first start >>>>> airflow with commands >>>>> $ airflow standalone >>>>> by making these two changes in airflow.cfg >>>>> *from * >>>>> executor = SequentialExecutor >>>>> >>>>> *to*executor = LocalExecutor >>>>> *&* >>>>> *from* >>>>> sql_alchemy_conn = sqlite:////home/zahid/airflow/airflow.db >>>>> *to* >>>>> sql_alchemy_conn = mysql+mysqldb://airflow_user:airflow_pass@localhost >>>>> :3306/airflow_db >>>>> >>>>> >>>>> *What about* variable sql_engine_collation_for_ids=utf8mb3_bin >>>>> mentioned next on same page >>>>> >>>>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>>>> should I set that in /etc/mysql/my.cnf also ? >>>>> >>>>> In the airflow.cfg the default setting is sql_engine_encoding = utf-8 >>>>> Should I change from sql_engine_encoding = utf-8 in airflow,cfg to >>>>> sql_engine_encoding = utf8mb4 in airflow.cfg if I am changing >>>>> sql_engine_collation_for_ids=utf8mb3_bin in /etc/mysql/my.cnf * ?* >>>>> >>>>> >>>>> Regards >>>>> Zahid >>>>> >>>>> http://www.frontbutton.com/ >>>>> ¯\_(ツ)_/¯ >>>>> ♡۶♡ ۶♡۶ >>>>> >>>>> <http://www.backbutton.co.uk> >>>>> >>>>> >>>>> On Fri, 22 Dec 2023 at 19:33, Lars Winderling < >>>>> lars.winderl...@posteo.de> wrote: >>>>> >>>>>> Hi Zahid, >>>>>> >>>>>> the additional MySQL config for the timestamps should go into a >>>>>> config file for MySQL. By default, it is called my.cnf. But you can pass >>>>>> it >>>>>> as a cli option, when starting the mysql daemon (as cited in your >>>>>> message). >>>>>> As an option to the mysql daemon mysqld, it defines the behavior of only >>>>>> the mysql daemon, so you cannot set it on client side. The airflow.cfg is >>>>>> the wrong place. It might work, but maybe in the long run not as expected >>>>>> due to quirks with MySQL not following the SQL standard very closely. >>>>>> Best, Lars >>>>>> >>>>>> >>>>>> On 22 December 2023 15:07:07 CET, Zahid Rahman <zahidr1...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> It was unclear from this page >>>>>>> >>>>>>> https://airflow.apache.org/docs/apache-airflow/2.8.0/howto/set-up-database.html#setting-up-a-mysql-database >>>>>>> which is a link from the airflow home when airflow first started. >>>>>>> that I need to update the *airflow.cfg* to configure the mysql >>>>>>> database like so. >>>>>>> >>>>>>> # sql_alchemy_conn = sqlite:////home/zahid/airflow/airflow.db >>>>>>> sql_alchemy_conn = >>>>>>> mysql+mysqldb://airflow_user:airflow_pass@localhost:3306/airflow_db >>>>>>> >>>>>>> Perhaps you may wish to update the page. >>>>>>> >>>>>>> ALSO >>>>>>> I did not make any change to file "my.cnf" nor do I have one that I >>>>>>> found in the ~/airflow as described below. >>>>>>> >>>>>>> "We rely on more strict ANSI SQL settings for MySQL in order to have >>>>>>> sane defaults. Make sure to have specified >>>>>>> explicit_defaults_for_timestamp=1 option under [mysqld] section in >>>>>>> your my.cnf file. You can also activate these options with the >>>>>>> --explicit-defaults-for-timestamp switch passed to mysqld >>>>>>> executable" >>>>>>> >>>>>>> This instruction did confuse me a bit at first until I took the >>>>>>> guess to make change in the airflow.cfg and *it worked!* >>>>>>> >>>>>>> Regards >>>>>>> Zahid >>>>>>> >>>>>>> http://www.frontbutton.com/ >>>>>>> ¯\_(ツ)_/¯ >>>>>>> ♡۶♡ ۶♡۶ >>>>>>> >>>>>>> <http://www.backbutton.co.uk> >>>>>>> >>>>>>
