I think the proper solution would be to delete the V004_ migration. This migration originally landed in the 0.14.0 release. Removing this migration means an upgrade directly from 0.13.0 to 0.18.0 would not be possible, but I suspect it wouldn't be possible for other reasons as well (i.e. too large of a jump in mesos versions for a single upgrade).
Unfortunately I didn't account for this scenario when I wrote the MigrationManager[1]. Specifically, the way it handles rollbacks is by comparing the list of applied migrations from the CHANGELOG table (descending by id) to the migrations found on the classpath. If it finds a migration in the CHANGELOG table that doesn't exist on the classpath, it will invoke the downgrade script that was stored in the CHANGELOG table when the migration was first applied. Thankfully as soon as it finds a migration that has been applied and still exists, it assumes no rollback and short circuits the process. So, in practice, as long as the V005_ migration still exists on the classpath, deleting the V004_ migration shouldn't have an impact. That said, it would still make me nervous to leave that gap there, due to the potential for unforeseen consequences and the possibly disastrous nature of inadvertently rolling back a DB change that shouldn't be rolled back. Perhaps a better solution is to just update the V004_ migration so that its `getUpScript` and `getDownScript` methods are just no-op SQL statements. That way people starting fresh after this change will still have a record of the migration being run, it will just have been a no-op. I think we'd need to think the various scenarios through carefully though (clean install, roll forward between releases, roll back between releases...), to ensure there are no cases I'm missing. [1] https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/storage/db/MigrationManagerImpl.java On Wed, Feb 15, 2017 at 12:20 PM, Nicolas Donatucci <ndonatu...@medallia.com > wrote: > Hi, I'm currently working on removing the deprecated fields numCpus, ramMb > and diskMb from the TaskConfig. > > After removing them from the schema.sql and adding the corresponding DB > migration, when building, there are four tests that fail. > > These are: > > - testMigrateToDBTaskStore > - testMigrateFromDBTaskStore > - testNoDBTaskStore > - testDBTaskStore > > They all throw a Storage Exception and the stacktrace (for > testMigrateToDBTaskStore) is: > > 0215 15:17:53.635 [main, h2database] jdbc[4] exception > org.h2.jdbc.JdbcSQLException: Column "T.NUM_CPUS" not found; SQL statement: > MERGE INTO task_resource(task_config_id, type_id, value) > KEY(task_config_id, type_id, value) SELECT t.id, rt.id, t.num_cpus FROM > task_configs t JOIN resource_types rt ON rt.name = 'CPUS' > [42122-193] > at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.message.DbException.get(DbException.java:179) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.message.DbException.get(DbException.java:155) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:147) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.command.dml.Select.prepare(Select.java:853) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.command.dml.Merge.prepare(Merge.java:260) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.command.Parser.prepareCommand(Parser.java:259) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.engine.Session.prepareLocal(Session.java:561) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.engine.Session.prepareCommand(Session.java:502) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1203) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.jdbc.JdbcStatement.executeInternal(JdbcStatement.java:170) > ~[h2-1.4.193.jar:1.4.193] > at org.h2.jdbc.JdbcStatement.execute(JdbcStatement.java:158) > ~[h2-1.4.193.jar:1.4.193] > at > org.apache.ibatis.jdbc.ScriptRunner.executeStatement( > ScriptRunner.java:235) > [mybatis-3.4.1.jar:3.4.1] > at org.apache.ibatis.jdbc.ScriptRunner.handleLine(ScriptRunner.java:208) > [mybatis-3.4.1.jar:3.4.1] > at > org.apache.ibatis.jdbc.ScriptRunner.executeLineByLine( > ScriptRunner.java:140) > [mybatis-3.4.1.jar:3.4.1] > at org.apache.ibatis.jdbc.ScriptRunner.runScript(ScriptRunner.java:107) > [mybatis-3.4.1.jar:3.4.1] > at > org.apache.ibatis.migration.operations.UpOperation. > operate(UpOperation.java:50) > [mybatis-migrations-3.2.0.jar:3.2.0] > at > org.apache.aurora.scheduler.storage.db.MigrationManagerImpl.migrate( > MigrationManagerImpl.java:65) > [production/:na] > at > org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl$ > 1.restoreFromSnapshot(SnapshotStoreImpl.java:176) > [production/:na] > at > org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl. > lambda$applySnapshot$1(SnapshotStoreImpl.java:498) > [production/:na] > at > org.apache.aurora.scheduler.storage.Storage$MutateWork$ > NoResult.apply(Storage.java:152) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.Storage$MutateWork$ > NoResult.apply(Storage.java:147) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage. > transactionedWrite(DbStorage.java:162) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage$$ > EnhancerByGuice$$df3385de.CGLIB$transactionedWrite$4(<generated>) > ~[guice-3.0.jar:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage$$ > EnhancerByGuice$$df3385de$$FastClassByGuice$$4b048fd7.invoke(<generated>) > ~[guice-3.0.jar:na] > at > com.google.inject.internal.cglib.proxy.$MethodProxy. > invokeSuper(MethodProxy.java:228) > ~[guice-3.0.jar:na] > at > com.google.inject.internal.InterceptorStackCallback$ > InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) > ~[guice-3.0.jar:na] > at > org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke( > TransactionalMethodInterceptor.java:101) > ~[mybatis-guice-3.7.jar:3.7] > at > com.google.inject.internal.InterceptorStackCallback$ > InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) > ~[guice-3.0.jar:na] > at > com.google.inject.internal.InterceptorStackCallback.intercept( > InterceptorStackCallback.java:52) > ~[guice-3.0.jar:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage$$ > EnhancerByGuice$$df3385de.transactionedWrite(<generated>) > ~[guice-3.0.jar:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage.lambda$ > write$0(DbStorage.java:176) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.db.DbModule$1$1. > closeDuring(DbModule.java:213) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:174) > ~[production/:na] > at > org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot( > SnapshotStoreImpl.java:494) > [production/:na] > at > org.apache.aurora.scheduler.storage.log.SnapshotStoreImpl.applySnapshot( > SnapshotStoreImpl.java:77) > [production/:na] > at > org.apache.aurora.scheduler.storage.log.SnapshotStoreImplIT. > testMigrateToDBTaskStore(SnapshotStoreImplIT.java:144) > ~[test/:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_112] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > ~[na:1.8.0_112] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_112] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall( > FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run( > ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively( > FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod. > evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.BlockJUnit4ClassRunner.runChild( > BlockJUnit4ClassRunner.java:78) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.BlockJUnit4ClassRunner.runChild( > BlockJUnit4ClassRunner.java:57) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > ~[junit-4.12.jar:4.12] > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > ~[junit-4.12.jar:4.12] > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > ~[junit-4.12.jar:4.12] > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs( > JUnit4IdeaTestRunner.java:117) > ~[junit-rt.jar:na] > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs( > JUnit4IdeaTestRunner.java:42) > ~[junit-rt.jar:na] > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart( > JUnitStarter.java:262) > ~[junit-rt.jar:na] > at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84) > ~[junit-rt.jar:na] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_112] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > ~[na:1.8.0_112] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_112] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112] > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) > ~[idea_rt.jar:na] > > So the problem seems to be that the migration V004 is running, and it tries > to read the no-longer existing values. > Removing the call to "migration()" in the V004 getUpScript() solves the > issue. > > Has anyone got an idea on what can be done with this? > Thanks! >
