Hi,
I was trying to switch from Sqlite/SequentialExecutor over to
MariaDB/LocalExecutor.
However, the ``airflow initdb`` step failed, with the message:
``Specified key was too long; max key length is 767 bytes``
Turns out that the default charset in MariaDB is ``utf8mb4``, and the
default max. keylength (**) is 767 bytes. The field ``dag_id`` is
defined as ``VARCHAR(250)``, 250 x 4 = 1000 > 767, hence the problem.
There are three solutions (workarounds) to this:
(1)
change the encoding to ``utf8``, which is not recommended however [3]
(2)
use a recent MariaDB/MySQL version via Docker, which is fine on my
machine, but cannot be used in production.
(3)
turn on ``innodb_large_prefix`` and related configs [4]. However, this
requires the option "ROW_FORMAT=DYNAMIC" (and maybe also
"ENGINE=InnoDB") to be set on each CREATE statement.
SqlAlchemy supports this [5], but the question is whether Airflow has
some mechanics built in to pass this option in via some config? Or did I
miss something / is there a fix already?
BTW: I am using MariaDB v10.1.30 (provided by my system)
Thanks in advance,
Andreas
(**) MariaDB < 10.2.2 [1] // MySQL < 5.7.7 [2]
[1]
https://mariadb.com/kb/en/library/xtradbinnodb-server-system-variables/#innodb_large_prefix
[2]
https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix
[3] https://stackoverflow.com/a/766996/6699237
[4] https://wiki.archlinux.org/index.php/MySQL#Increase_character_limit
[5]
http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#create-table-arguments-including-storage-engines
--
Andreas Koeltringer
Mail: [email protected]
