hi list,
we try to use HiveToMySqlTransfer to write the hive data into MySQL. but
there was an auto increment id in our table.
like this:
> create table users_simple (`id` int NOT NULL AUTO_INCREMENT,
`user_id` bigint NOT NULL, PRIMARY KEY(`id`));
in airflow DAG we use HiveToMySqlTransfer:
t = HiveToMySqlTransfer(
mysql_conn_id='data_pipeline_mysql',
task_id='hive_to_mysql_check',
hiveserver2_conn_id='hiveserver2_default',
sql="""
SELECT login
FROM staging.users_simple limit 100
""",
mysql_table='users_simple',
dag=dag)
when run the code above, we met the following exception:
> _mysql_exceptions.OperationalError: (1136, "Column count doesn't match
value count at row 1")
I believe the airflow translate the task as the SQL statement:
> insert into users_simple(id, user_id) values (login)
so the solution is how we pass the exactly columns to dbapi_hook.py
https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/hooks/dbapi_hook.py#L187
via HiveToMySqlTransfer ?
--
*Ric Dong*
