lintingbin opened a new issue, #3261:
URL: https://github.com/apache/amoro/issues/3261
### What happened?
Upgrading the iceberg version can cause AMS startup to fail when processes
are present.
### Affects Versions
0.7.0
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
Upgrading the iceberg version while retaining the process prior to the
upgrade can cause AMS to fail to start when using the higher version of iceberg.
### Relevant log output
```shell
2024-10-16 02:33:18,933 ERROR [main]
[org.apache.amoro.server.AmoroServiceContainer] [] - AMS start error
org.apache.amoro.server.exception.PersistenceException:
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.lang.IllegalArgumentException:
deserialization error
### The error may exist in
org/apache/amoro/server/persistence/mapper/OptimizingMapper.java (best guess)
### The error may involve
org.apache.amoro.server.persistence.mapper.OptimizingMapper.selectTaskRuntimes
### The error occurred while handling results
### SQL: SELECT process_id, task_id, retry_num, table_id, partition_data,
create_time, start_time, end_time, status, fail_reason, optimizer_token,
thread_id, rewrite_output, metrics_summary, properties FROM task_runtime WHERE
table_id = ? AND process_id = ?
### Cause: java.lang.IllegalArgumentException: deserialization error
at
org.apache.amoro.server.exception.AmoroRuntimeException.wrap(AmoroRuntimeException.java:141)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.PersistentBase.getAs(PersistentBase.java:92)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.optimizing.OptimizingQueue.access$500(OptimizingQueue.java:73)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.loadTaskRuntimes(OptimizingQueue.java:674)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.<init>(OptimizingQueue.java:406)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.optimizing.OptimizingQueue.initTableRuntime(OptimizingQueue.java:116)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_422]
at
org.apache.amoro.server.optimizing.OptimizingQueue.<init>(OptimizingQueue.java:109)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.DefaultOptimizingService.lambda$loadOptimizingQueues$0(DefaultOptimizingService.java:142)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_422]
at
org.apache.amoro.server.DefaultOptimizingService.loadOptimizingQueues(DefaultOptimizingService.java:132)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.DefaultOptimizingService.access$300(DefaultOptimizingService.java:88)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.DefaultOptimizingService$TableRuntimeHandlerImpl.initHandler(DefaultOptimizingService.java:462)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.table.RuntimeHandlerChain.initialize(RuntimeHandlerChain.java:60)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.table.DefaultTableService.initialize(DefaultTableService.java:324)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.AmoroServiceContainer.startService(AmoroServiceContainer.java:158)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.AmoroServiceContainer.main(AmoroServiceContainer.java:116)
[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.lang.IllegalArgumentException:
deserialization error
### The error may exist in
org/apache/amoro/server/persistence/mapper/OptimizingMapper.java (best guess)
### The error may involve
org.apache.amoro.server.persistence.mapper.OptimizingMapper.selectTaskRuntimes
### The error occurred while handling results
### SQL: SELECT process_id, task_id, retry_num, table_id, partition_data,
create_time, start_time, end_time, status, fail_reason, optimizer_token,
thread_id, rewrite_output, metrics_summary, properties FROM task_runtime WHERE
table_id = ? AND process_id = ?
### Cause: java.lang.IllegalArgumentException: deserialization error
at
org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:156)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
~[mybatis-3.5.15.jar:3.5.15]
at com.sun.proxy.$Proxy48.selectTaskRuntimes(Unknown Source) ~[?:?]
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.lambda$loadTaskRuntimes$13(OptimizingQueue.java:677)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.PersistentBase.getAs(PersistentBase.java:90)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
... 15 more
Caused by: java.lang.IllegalArgumentException: deserialization error
at
org.apache.amoro.utils.SerializationUtil.simpleDeserialize(SerializationUtil.java:68)
~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.converter.Object2ByteArrayConvert.getResult(Object2ByteArrayConvert.java:61)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getPropertyMappingValue(DefaultResultSetHandler.java:530)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyPropertyMappings(DefaultResultSetHandler.java:495)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:418)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:366)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:337)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:310)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:202)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:66)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:80)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:65)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:336)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:158)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:110)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:90)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:154)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
~[mybatis-3.5.15.jar:3.5.15]
at com.sun.proxy.$Proxy48.selectTaskRuntimes(Unknown Source) ~[?:?]
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.lambda$loadTaskRuntimes$13(OptimizingQueue.java:677)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.PersistentBase.getAs(PersistentBase.java:90)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
... 15 more
Caused by: java.io.InvalidClassException: org.apache.iceberg.BaseFile; local
class incompatible: stream classdesc serialVersionUID = -5355849892016662001,
local class serialVersionUID = 4493876333706690896
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
~[?:1.8.0_422]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
~[?:1.8.0_422]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
~[?:1.8.0_422]
at
org.apache.amoro.utils.SerializationUtil.simpleDeserialize(SerializationUtil.java:65)
~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.converter.Object2ByteArrayConvert.getResult(Object2ByteArrayConvert.java:61)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getPropertyMappingValue(DefaultResultSetHandler.java:530)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyPropertyMappings(DefaultResultSetHandler.java:495)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:418)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:366)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:337)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:310)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:202)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:66)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:80)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:65)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:336)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:158)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:110)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:90)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:154)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
~[mybatis-3.5.15.jar:3.5.15]
at
org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
~[mybatis-3.5.15.jar:3.5.15]
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
~[mybatis-3.5.15.jar:3.5.15]
at com.sun.proxy.$Proxy48.selectTaskRuntimes(Unknown Source) ~[?:?]
at
org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.lambda$loadTaskRuntimes$13(OptimizingQueue.java:677)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
at
org.apache.amoro.server.persistence.PersistentBase.getAs(PersistentBase.java:90)
~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
... 15 more
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
