19922577513 opened a new issue, #3364: URL: https://github.com/apache/paimon/issues/3364
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version paimon-flink-1.17-0.8.jar paimon-flink-action-0.8.0.jar paimon-hive-connector-2.3-0.8.0.jar ### Compute Engine flink version -> 1.17.2 hadoop -> 3.3.5 ### Minimal reproduce step 1、create test table CREATE TABLE `data_management`.`sql_template_info_paimon` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id', `data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive', `table_name` varchar(100) NOT NULL COMMENT '当前操作的表名', `chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名', `dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度', `table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层', `is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表', `table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型', `table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级', `table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别', `partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存', `partition_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存', `update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率', `remark` longtext COMMENT '备注', `old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开', `define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开', `heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度', `principal` varchar(60) NOT NULL COMMENT '当前表负责人', `process_id` int(10) DEFAULT NULL COMMENT '工作流id', `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间', `special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息' PRIMARY KEY (`id`), UNIQUE KEY `uniq_n` (`table_name`), KEY `idx_process_id` (`process_id`) ) ENGINE=InnoDB AUTO_INCREMENT=68318 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表' 2、Synchronize tasks through "mysql_sync_database" sudo ./bin/flink run --allowNonRestoredState /home/q/module/flink/flink-1.17.2/lib/paimon-flink-action-0.8.0.jar mysql_sync_database --warehouse viewfs://qunarcluster/user/tujiadev/home/data/paimon/warehouse --type_mapping tinyint1-not-bool,to-nullable,char-to-string,bigint-unsigned-to-bigint --database tujia_ods --mode combined --merge_shards false --table_prefix ods_ --table_suffix _paimon --mysql_conf hostname=xx --mysql_conf username=xx --mysql_conf password=xx --mysql_conf database-name='data_management' --mysql_conf port=3310 --mysql_conf server-id=6289-6298 --mysql_conf scan.incremental.snapshot.chunk.size=80960 --table_conf bucket=10 --table_conf sink.parallelism=1 --table_conf file.format=parquet --table_conf file.compression=snappy --table_conf scan.infer-parallelism=false --table_conf scan.parallelism=100 --including_tables 'mail_tasks' --catalog_conf metastore=hive --catalog_conf uri='thrift://l-hiveserver4.data.cn2:9083,thrift://l-hiveserver5.data.cn2:9083,thrift://l- hiveserver6.data.cn2:9083,thrift://l-hiveserver7.data.cn2:9083,thrift://l-hiveserver8.data.cn2:9083' 3、The sql submitted by mysql is as follows: ①、 "CREATE TABLE `data_management`.`_sql_template_info_paimon_new` (换行 `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',换行 `data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive',换行 `table_name` varchar(100) NOT NULL COMMENT '当前操作的表名',换行 `chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名',换行 `dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度',换行 `table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层',换行 `is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表',换行 `table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型',换行 `table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级',换行 `table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别',换行 `partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存',换行 `partit ion_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存',换行 `update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率',换行 `remark` longtext COMMENT '备注',换行 `old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开',换行 `define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开',换行 `heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度',换行 `principal` varchar(60) NOT NULL COMMENT '当前表负责人',换行 `process_id` int(10) DEFAULT NULL COMMENT '工作流id',换行 `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',换行 `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',换行 `special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息',换行 PRIMARY KEY (`id`),换行 UNIQUE KEY `uniq_n` (`table_name`),换� �� KEY `idx_process_id` (`process_id`)换行) ENGINE=InnoDB AUTO_INCREMENT=68315 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表'", ②、 "ALTER TABLE `data_management`.`_sql_template_info_paimon_new` ADD COLUMN paimon_field VARCHAR(20) DEFAULT '' NOT NULL COMMENT '辨别'" ③、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_del` AFTER DELETE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW DELETE IGNORE FROM `data_management`.`_sql_template_info_paimon_new` WHERE `data_management`.`_sql_template_info_paimon_new`.`id` <=> OLD.`id`" ④、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_upd` AFTER UPDATE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_ level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)" ⑤、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_ins` AFTER INSERT ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_ level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)" ⑥、 "ANALYZE TABLE `data_management`.`_sql_template_info_paimon_new` /* pt-online-schema-change */" "RENAME TABLE `data_management`.`sql_template_info_paimon` TO `data_management`.`_sql_template_info_paimon_old`" ⑦、"DROP TABLE IF EXISTS `data_management`.`_sql_template_info_paimon_old`" ⑧、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_del`" ⑨、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_upd`" ⑩、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_ins`" 4、Insert a random piece of data into this table 5、After the binlog consumes the current insert data, you can see that checkpoint is in the IN_PROGRESS state indefinitely ### What doesn't meet your expectations? After the synchronized mysql table structure changes, the paimon table cannot capture the change, and most importantly, the entire task is stuck ### Anything else? The taskmanager prints two types of logs. One is the error log caused by the DDL-related binlog. The other is the info log where the table has been waiting for the field schame to update 1、Invalid historyRecord, because tableChanges should contain exactly 1 item. 2、Field [???] not found. Waiting for schema update. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
