Hello, Can you try to restore the secondary db using the latest primary backup and see if that fixes the issue.
On Friday, September 25, 2020 at 3:18:44 PM UTC+5:30 [email protected] wrote: > I tried to fix liquibase exception error by doing the following: > > 1. stopping gocd on both nodes > 2. stopping the standby DB > 3. on the primary clearing the lock by running UPDATE > DATABASECHANGELOGLOCK SET LOCKED=FALSE, LOCKGRANTED=null, LOCKEDBY=null; > 4. stop the primary DB too > 5. start the DB on secondary > 6. start the db on primary > 7. start gocd on secondary > 8. start gocd on primary > > after a while, the secondary GoCD crashed again wit the following errors: > > on go-server-wrapper.log: > > INFO | jvm 1 | 2020/07/09 14:35:40 | WARNING: An illegal reflective > access operation has occurred > INFO | jvm 1 | 2020/07/09 14:35:40 | WARNING: Illegal reflective > access by org.bouncycastle.jcajce.provider.drbg.DRBG > (onejar:lib/bcprov-jdk15on-1.59.jar) to constructor > sun.security.provider.Sun() > INFO | jvm 1 | 2020/07/09 14:35:40 | WARNING: Please consider > reporting this to the maintainers of > org.bouncycastle.jcajce.provider.drbg.DRBG > INFO | jvm 1 | 2020/07/09 14:35:40 | WARNING: Use > --illegal-access=warn to enable warnings of further illegal reflective > access operations > INFO | jvm 1 | 2020/07/09 14:35:40 | WARNING: All illegal access > operations will be denied in a future release > INFO | jvm 1 | 2020/07/09 14:35:49 | ERROR: Failed to start GoCD > server. Please check the logs. > INFO | jvm 1 | 2020/07/09 14:35:49 | java.lang.RuntimeException: > Failed to start GoCD server. > ... > ... > INFO | jvm 1 | 2020/07/09 14:35:49 | Caused by: > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'webhookControllerV1' defined in URL > [jar:file:/var/lib/go-server/work/jetty-192.168.0.48-8153-cruise.war-_go-any-/webapp/WEB-INF/lib/api-webhook-v1-20.1.0-11114.jar!/com/thoughtworks/go/apiv1/webhook/WebhookControllerV1.class]: > > Unsatisfied dependency expressed through constructor parameter 0; nested > exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'sqlSessionFactoryBean' defined in file > [/var/lib/go-server/work/jetty-192.168.0.48-8153-cruise.war-_go-any-/webapp/WEB-INF/classes/com/thoughtworks/go/server/database/SqlSessionFactoryBean.class]: > > Unsatisfied dependency expressed through constructor parameter 0; nested > exception is org.springframework.beans.factory.BeanCreationException: Error > creating bean with name 'databaseStrategy' defined in file > [/var/lib/go-server/work/jetty-192.168.0.48-8153-cruise.war-_go-any-/webapp/WEB-INF/classes/com/thoughtworks/go/server/database/DatabaseStrategy.class]: > > Bean instantiation via constructor failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to instantiate > [com.thoughtworks.go.server.database.DatabaseStrategy]: Constructor threw > exception; nested exception is java.lang.RuntimeException: Failed loading > database provider [com.thoughtworks.go.postgresql.PostgresqlDatabasei] > > > than again on go-server.log > > 2020-09-25 10:02:13,350 ERROR [WrapperJarAppMain] DatabaseMigrator:65 - > Unable to create database upgrade script for database. The problem was: > liquibase.exception.DatabaseException: Error executing SQL UPDATE > public.databasechangeloglock SET LOCKED = TRUE, LOCKEDBY = > 'go02-000-021-0', LOCKGRANTED = '2020-09-25 10:02:13.346' WHERE ID = 1 AND > LOCKED = FALSE: ERROR: cannot execute UPDATE in a read-only transaction. > The cause was: Error executing SQL UPDATE public.databasechangeloglock SET > LOCKED = TRUE, LOCKEDBY = 'go02-000-021-0', LOCKGRANTED = '2020-09-25 > 10:02:13.346' WHERE ID = 1 AND LOCKED = FALSE: ERROR: cannot execute UPDATE > in a read-only transaction > liquibase.exception.LockException: liquibase.exception.DatabaseException: > Error executing SQL UPDATE public.databasechangeloglock SET LOCKED = TRUE, > LOCKEDBY = 'go02-000-021-0', LOCKGRANTED = '2020-09-25 10:02:13.346' WHERE > ID = 1 AND LOCKED = FALSE: ERROR: cannot execute UPDATE in a read-only > transaction > > > The primary is running fine, the dashboard is working and all the pipeline > are in place. > > Does anybody have any idea ? > Thanks in advance > > On Friday, September 25, 2020 at 9:20:45 AM UTC+2 Luca G. Soave wrote: > >> >> >> We have two GoCD 20.7 servers with business continuity and Postgres >> 9.6.10 backend. >> >> Lately we re-started the standby and the master DB/GoCD but soon after, >> the standby crashed with the following error: >> >> INFO | jvm 1 | 2020/09/22 12:24:40 | Unable to create database >> upgrade script for database. The problem was: >> liquibase.exception.DatabaseException: Error executing SQL UPDATE >> public.databasechangeloglock SET LOCKED = TRUE, LOCKEDBY = >> 'go02-000-021-0', LOCKGRANTED = '2020-09-22 12:24:40.632' WHERE ID = 1 AND >> LOCKED = FALSE: ERROR: cannot execute UPDATE in a read-only transaction. >> The cause was: Error executing SQL UPDATE public.databasechangeloglock SET >> LOCKED = TRUE, LOCKEDBY = 'go02-000-021-0', LOCKGRANTED = '2020-09-22 >> 12:24:40.632' WHERE ID = 1 AND LOCKED = FALSE: ERROR: cannot execute UPDATE >> in a read-only transaction >> >> INFO | jvm 1 | 2020/09/22 12:24:40 | >> liquibase.exception.LockException: liquibase.exception.DatabaseException: >> Error executing SQL UPDATE public.databasechangeloglock SET LOCKED = TRUE, >> LOCKEDBY = 'go02-000-021-0', LOCKGRANTED = '2020-09-22 12:24:40.632' WHERE >> ID = 1 AND LOCKED = FALSE: ERROR: cannot execute UPDATE in a read-only >> transaction >> >> The master is up & running and the admin dashboard is working. >> >> Do you have any help ? >> >> Thanks in advance >> >> Luca >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/ad9ae672-ecc8-4559-b679-5bdb33657ce7n%40googlegroups.com.
