GangYang-HX opened a new issue, #4949: URL: https://github.com/apache/paimon/issues/4949
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version paimon-0.9.0 paimon-1.0.0 ### Compute Engine Flink-1.18.1 ### Minimal reproduce step Pamon table creation script： `CREATE TABLE `paimon_test.user_behavior_fusion_di_test_13`( `seq` int COMMENT 'from deserializer', `meta_id` int COMMENT 'from deserializer', `service_id` string COMMENT 'from deserializer', `client_timestamp` bigint COMMENT 'from deserializer', `server_timestamp` bigint COMMENT 'from deserializer', `props` map<string,string> COMMENT 'from deserializer', `channel_name` string COMMENT 'from deserializer', `os_name` string COMMENT 'from deserializer', `version_no` string COMMENT 'from deserializer', `session_id` string COMMENT 'from deserializer', `device_id` string COMMENT 'from deserializer', `device_name` string COMMENT 'from deserializer', `device_type_name` string COMMENT 'from deserializer', `user_id` bigint COMMENT 'from deserializer', `app_id` int COMMENT 'from deserializer', `bid` int COMMENT 'from deserializer', `extra_attribute` map<string,string> COMMENT 'from deserializer', `client_send_timestamp` bigint COMMENT 'from deserializer', `insert_timestamp` timestamp COMMENT 'from deserializer', `uncommon_map` map<string,string> COMMENT 'from deserializer', `random_column` int COMMENT 'from deserializer', `data_version` int COMMENT 'from deserializer', `location_code` bigint COMMENT 'from deserializer', `duration_time` bigint COMMENT 'from deserializer', `device_id_code` bigint COMMENT 'from deserializer', `is_lock_exposed` boolean COMMENT 'from deserializer', `from_back` int COMMENT 'from deserializer') PARTITIONED BY ( `dt` string COMMENT '日期, yyyyMMdd', `event_id` string COMMENT '事件类型') ROW FORMAT SERDE 'org.apache.paimon.hive.PaimonSerDe' STORED BY 'org.apache.paimon.hive.PaimonStorageHandler' WITH SERDEPROPERTIES ( 'serialization.format'='1') TBLPROPERTIES ( 'bucket'='-1', 'bucketing_version'='2', 'file-index.bitmap.columns'='app_id,meta_id,bid', 'file.format'='parquet', 'num-sorted-run.stop-trigger'='30', 'parquet.compression'='zstd', 'partition.expiration-check-interval'='1d', 'partition.expiration-time'='365d', 'partition.timestamp-formatter'='yyyyMMdd', 'partition.timestamp-pattern'='$dt', 'snapshot.expire.limit'='8', 'snapshot.num-retained.min'='16', 'snapshot.time-retained'='6h', 'sort-spill-threshold'='10', 'target-file-size'='1GB', 'transient_lastDdlTime'='1737100343', 'write-buffer-size'='512MB', 'write-manifest-cache'='1GB')`。 The task will generally succeed in the first checkpoint. In terms of performance, it will be written successfully when there is no partition information, and an exception will be reported later. ### What doesn't meet your expectations? `2025-01-20 11:12:02 java.lang.RuntimeException: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?) at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:95) at org.apache.paimon.metastore.AddPartitionCommitCallback.retry(AddPartitionCommitCallback.java:76) at org.apache.paimon.operation.FileStoreCommitImpl.lambda$filterCommitted$0(FileStoreCommitImpl.java:247) at java.util.ArrayList.forEach(ArrayList.java:1259) at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:247) at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:244) at org.apache.paimon.flink.sink.StoreCommitter.filterAndCommit(StoreCommitter.java:119) at org.apache.paimon.flink.sink.Committer.filterAndCommit(Committer.java:60) at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.recover(RestoreAndFailCommittableStateManager.java:82) at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.initializeState(RestoreAndFailCommittableStateManager.java:77) at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:142) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106) at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:753) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:728) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) at java.lang.Thread.run(Thread.java:750) Caused by: MetaException(message:Exception thrown when executing query : SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`PART_ID` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `A0`.`PART_NAME` = ? AND `C0`.`CTLG_NAME` = ?) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:2488) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:2475) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:695) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) at com.sun.proxy.$Proxy27.add_partitions(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) at com.sun.proxy.$Proxy28.add_partitions(Unknown Source) at org.apache.paimon.hive.HiveMetastoreClient.lambda$addPartitions$3(HiveMetastoreClient.java:107) at org.apache.paimon.client.ClientPool$ClientPoolImpl.lambda$execute$0(ClientPool.java:80) at org.apache.paimon.client.ClientPool$ClientPoolImpl.run(ClientPool.java:68) at org.apache.paimon.client.ClientPool$ClientPoolImpl.execute(ClientPool.java:77) at org.apache.paimon.hive.pool.CachedClientPool.execute(CachedClientPool.java:139) at org.apache.paimon.hive.HiveMetastoreClient.addPartitions(HiveMetastoreClient.java:107) at org.apache.paimon.metastore.AddPartitionCommitCallback.addPartitions(AddPartitionCommitCallback.java:88) ... 22 more ` ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
