vanphuoc3012 opened a new issue, #3136: URL: https://github.com/apache/amoro/issues/3136
### What happened? I see lot of this exception in logs: amoro-ams ### Error updating database. Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "optimizing_task_quota_pkey" amoro-ams Detail: Key (process_id, task_id, retry_num)=(1724644992030, 86, 0) already exists. I check table optimizing_task_quota and delete record which process_id = 1724644992030, exception disappear This may cause amoro cannot process any new table optimize task ### Affects Versions 0.7.0-incubating ### What table formats are you seeing the problem on? Iceberg ### What engines are you seeing the problem on? Spark ### How to reproduce _No response_ ### Relevant log output ```shell > amoro-ams 2024-08-26 05:29:56,368 ERROR [thrift-server-optimize-manager-43] [org.apache.amoro.server.TableManagementService] [] - Thrift service:DefaultOptimizingService.completeTask execute failed > amoro-ams org.apache.amoro.server.exception.PersistenceException: java.lang.IllegalStateException: session is rolling back, can not execute operation > amoro-ams at org.apache.amoro.server.exception.AmoroRuntimeException.wrap(AmoroRuntimeException.java:141) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:65) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.StatedPersistentBase.invokeConsistency(StatedPersistentBase.java:51) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.optimizing.TaskRuntime.complete(TaskRuntime.java:77) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.DefaultOptimizingService.completeTask(DefaultOptimizingService.java:234) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at sun.reflect.GeneratedMethodAccessor369.invoke(Unknown Source) ~[?:?] > amoro-ams at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_422] > amoro-ams at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_422] > amoro-ams at org.apache.amoro.server.utils.ThriftServiceProxy.invoke(ThriftServiceProxy.java:56) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at com.sun.proxy.$Proxy51.completeTask(Unknown Source) [?:?] > amoro-ams at org.apache.amoro.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:629) [amoro-core-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.api.OptimizingService$Processor$completeTask.getResult(OptimizingService.java:605) [amoro-core-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.shade.thrift.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40) [amoro-shade-thrift-0.20.0-0.7-SNAPSHOT.jar:0.20.0-0.7-SNAPSHOT] > amoro-ams at org.apache.amoro.shade.thrift.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) [amoro-shade-thrift-0.20.0-0.7-SNAPSHOT.jar:0.20.0-0.7-SNAPSHOT] > amoro-ams at org.apache.amoro.shade.thrift.org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:147) [amoro-shade-thrift-0.20.0-0.7-SNAPSHOT.jar:0.20.0-0.7-SNAPSHOT] > amoro-ams at org.apache.amoro.shade.thrift.org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:492) [amoro-shade-thrift-0.20.0-0.7-SNAPSHOT.jar:0.20.0-0.7-SNAPSHOT] > amoro-ams at org.apache.amoro.shade.thrift.org.apache.thrift.server.Invocation.run(Invocation.java:19) [amoro-shade-thrift-0.20.0-0.7-SNAPSHOT.jar:0.20.0-0.7-SNAPSHOT] > amoro-ams at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422] > amoro-ams at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422] > amoro-ams at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422] > amoro-ams Caused by: java.lang.IllegalStateException: session is rolling back, can not execute operation > amoro-ams at org.apache.amoro.shade.guava32.com.google.common.base.Preconditions.checkState(Preconditions.java:512) ~[amoro-shade-guava-32-32.1.1-jre-0.7-SNAPSHOT.jar:32.1.1-jre-0.7-SNAPSHOT] > amoro-ams at org.apache.amoro.server.persistence.NestedSqlSession.checkState(NestedSqlSession.java:85) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.NestedSqlSession.commit(NestedSqlSession.java:67) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:62) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams ... 18 more > amoro-ams 2024-08-26 05:29:56,368 WARN [thrift-server-optimize-manager-34] [org.apache.amoro.server.optimizing.OptimizingQueue] [] - iceberg-hive-catalog.hydra.user_notification_v2(tableId=15) failed to add task quota OptimizingTaskId(p > amoro-ams org.apache.amoro.server.exception.PersistenceException: org.apache.ibatis.exceptions.PersistenceException: > amoro-ams ### Error updating database. Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "optimizing_task_quota_pkey" > amoro-ams Detail: Key (process_id, task_id, retry_num)=(1724644992030, 87, 0) already exists. > amoro-ams ### The error may exist in org/apache/amoro/server/persistence/mapper/OptimizingMapper.java (best guess) > amoro-ams ### The error may involve org.apache.amoro.server.persistence.mapper.OptimizingMapper.insertTaskQuota-Inline > amoro-ams ### The error occurred while setting parameters > amoro-ams ### SQL: INSERT INTO optimizing_task_quota (process_id, task_id, retry_num, table_id, start_time, end_time, fail_reason) VALUES (?, ?, ?, ?, ?, ?, ?) > amoro-ams ### Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "optimizing_task_quota_pkey" > amoro-ams Detail: Key (process_id, task_id, retry_num)=(1724644992030, 87, 0) already exists. > amoro-ams at org.apache.amoro.server.exception.AmoroRuntimeException.wrap(AmoroRuntimeException.java:141) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.PersistentBase.doAs(PersistentBase.java:53) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.table.TableRuntime.addTaskQuota(TableRuntime.java:359) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.acceptResult(OptimizingQueue.java:435) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.optimizing.TaskRuntime.lambda$complete$0(TaskRuntime.java:104) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:1.8.0_422] > amoro-ams at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) [?:1.8.0_422] > amoro-ams at org.apache.amoro.server.persistence.PersistentBase.doAsTransaction(PersistentBase.java:61) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.persistence.StatedPersistentBase.invokeConsistency(StatedPersistentBase.java:51) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.optimizing.TaskRuntime.complete(TaskRuntime.java:77) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at org.apache.amoro.server.DefaultOptimizingService.completeTask(DefaultOptimizingService.java:234) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] > amoro-ams at sun.reflect.GeneratedMethodAccessor369.invoke(Unknown Source) ~[?:?] > amoro-ams at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_422] > amoro-ams at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_422] > amoro-ams at org.apache.amoro.server.utils.ThriftServiceProxy.invoke(ThriftServiceProxy.java:56) [amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating] ``` ### Anything else _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
