potiuk commented on code in PR #25938:
URL: https://github.com/apache/airflow/pull/25938#discussion_r955038767
##########
docs/apache-airflow/installation/upgrading.rst:
##########
@@ -55,16 +67,143 @@ Sample usage:
``airflow db upgrade --revision-range "e959f08ac86c:142555e44c17"``
-Migration best practices
-========================
+Handling migration problems
+===========================
+
+
+Wrong Encoding in MySQL database
+................................
+
+If you are using old Airflow 1.10 as a database created initially either
manually or with previous version of MySQL,
+depending on the original character set of your database, you might have
problems with migrating to a newer
+version of Airflow and your migration might fail with strange errors ("key
size too big", "missing indexes" etc).
+The next chapter describes how to fix the problem manually.
Review Comment:
Good choice (of Postgres). No I have not used any myself but I saw people
reporting successes using simple export/import scripts. Actually migrating out
of MySQL to postgres is such a popular thing that there are PLENTY tools out
there (and actually I would love to hear more concrete success stories from
people like you :).
I personally think that the best approach is to create the database using
Airlfow `db` cli and only migrate the data via export/import through .csv files
(after earlier cleanup of historcal data to decrease the size of the data to
migrate and speed it up). Migrating structure is prone to errors but the data
in both databases should be exactly the same.
Of course assuming the export from MySQL will use the right encoding - which
is not at all given. Unfortunately MySQL complicates encodings in so many
strange ways that it's very likely this is not gonna work first time. For
example when you use mysql client to connect to mysql server, the default
encoding used by the client is NOT the server encoding but your machine
encoding - so you can get weird encoding issues if encoding on your "terminal'
machine is different than the one in the server (now you might slowly
understand my hatred for encoding ~approach ~ mess in MySQL).
Airflow has suprisingly little number of tables, so this can be done
entirely by writing hand-written scripts, there is no need to use specialized
tools for that. The only thing that you have to remember is to do the import in
the right sequence due to foreign keys and likely speed up the import by
temporarily disabling the foreign keys while importing and re-enabling them
after.
##########
docs/apache-airflow/installation/upgrading.rst:
##########
@@ -55,16 +67,143 @@ Sample usage:
``airflow db upgrade --revision-range "e959f08ac86c:142555e44c17"``
-Migration best practices
-========================
+Handling migration problems
+===========================
+
+
+Wrong Encoding in MySQL database
+................................
+
+If you are using old Airflow 1.10 as a database created initially either
manually or with previous version of MySQL,
+depending on the original character set of your database, you might have
problems with migrating to a newer
+version of Airflow and your migration might fail with strange errors ("key
size too big", "missing indexes" etc).
+The next chapter describes how to fix the problem manually.
Review Comment:
Good choice (of Postgres). No I have not used any myself but I saw people
reporting successes using simple export/import scripts. Actually migrating out
of MySQL to postgres is such a popular thing that there are PLENTY tools out
there (and actually I would love to hear more concrete success stories from
people like you :).
I personally think that the best approach is to create the database using
Airlfow `db` cli and only migrate the data via export/import through .csv files
(after earlier cleanup of historcal data to decrease the size of the data to
migrate and speed it up). Migrating structure is prone to errors but the data
in both databases should be exactly the same.
Of course assuming the export from MySQL will use the right encoding - which
is not at all given. Unfortunately MySQL complicates encodings in so many
strange ways that it's very likely this is not gonna work first time. For
example when you use mysql client to connect to mysql server, the default
encoding used by the client is NOT the server encoding but your machine
encoding - so you can get weird encoding issues if encoding on your "terminal'
machine is different than the one in the server (now you might slowly
understand my hatred for encoding ~approach~ mess in MySQL).
Airflow has suprisingly little number of tables, so this can be done
entirely by writing hand-written scripts, there is no need to use specialized
tools for that. The only thing that you have to remember is to do the import in
the right sequence due to foreign keys and likely speed up the import by
temporarily disabling the foreign keys while importing and re-enabling them
after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
