Hi Lawrence
It looks like another user might has quite likely got to the bottom of this
problem: https://github.com/gocd/gocd/issues/13214
TL;DR - a library upgrade included in 24.3.0 (but not 24.2.0) changed the
random string generation Liquibase database upgrades/migrations rely upon
to a highly secure mechanism. In some environments without sufficient
entropy things would just block waiting for more entropy to generate random
numbers/strings.
While I cannot replicate this with MacOS/Colima containers, and I have not
seen an issue with our build/test Linux VMs/containers on AWS it's possible
other host or virtual environments would have issues, especially those
without much entropy and running Linux kernels < 5.6 on "real" hardware.
If someone wants to try and validate it is fixed, perhaps see if you have
the same issue with an experimental build like
gocdexperimental/gocd-server:v24.4.0-19635 which has upgraded libraries
that address the underlying problem.
-Chad
On Sat, Oct 19, 2024 at 10:29 AM Chad Wilson <ch...@thoughtworks.com> wrote:
> At a time when it is stuck, can you exec into the container and send kill
> -3 to the java process for the server? Then check the stdout and share the
> big thread dump (in normal chronological order please, not reversed 🙏)
>
> I need to see a thread dump to have an idea why it is stuck, as we don't
> have any steps to replicate this inside or outside a container.
>
> -Chad
>
> On Sat, Oct 19, 2024 at 7:24 AM Lawrence <lawrence.sp...@gmail.com> wrote:
>
>> Forgot to add details.
>>
>> Tried it on Windows 11 with Docker desktop v4.34.3 and Synology DSM
>> 7.2.1-69057 update 5 using container manager.
>>
>> They both have a log of:
>>
>> stdout wrapper | <-- Wrapper Stopped
>> stdout wrapper | TERM trapped. Shutting down.
>> stdout "jvm 1 | 2024-10-18 23:01:09,736 INFO [WrapperJarAppMain]
>> DatabaseMigrator:53 -
>> ************************************************************************"
>> stdout jvm 1 |
>> ************************************************************************
>> stdout "jvm 1 | 2024-10-18 23:01:09,735 INFO [WrapperJarAppMain]
>> DatabaseMigrator:53 - WARNING: Shutting down your server at this point will
>> lead to a database corruption. Please wait until the database upgrade
>> completes."
>> stdout jvm 1 | WARNING: Shutting down your server at this point will
>> lead to a database corruption. Please wait until the database upgrade
>> completes.
>> stdout "jvm 1 | 2024-10-18 23:01:09,734 INFO [WrapperJarAppMain]
>> DatabaseMigrator:53 -
>> ************************************************************************"
>> stdout jvm 1 |
>> ************************************************************************
>> stdout "jvm 1 | 2024-10-18 23:01:09,733 INFO [WrapperJarAppMain]
>> DatabaseMigrator:44 - Upgrading database, this might take a while depending
>> on the size of the database."
>> stdout "jvm 1 | 2024-10-18 23:01:09,186 WARN [WrapperJarAppMain]
>> ConnectionManager:117 - The file config/db.properties specified by
>> `go.db.config` does not exist."
>> stdout "jvm 1 | 2024-10-18 23:01:03,332 WARN [WrapperJarAppMain]
>> AbstractHandler:88 - No Server set for ResourceHandler@447a0fea{STOPPED}"
>> stdout "jvm 1 | 2024-10-18 23:01:03,212 WARN [WrapperJarAppMain]
>> Server:366 - ErrorPageMapper not supported for Server level Error Handling"
>> stdout "jvm 1 | 2024-10-18 23:01:03,060 INFO [WrapperJarAppMain]
>> JettyServer:194 - Configuring Jetty using /go-working-dir/config/jetty.xml"
>> stdout jvm 1 | Using classpath resource
>> `jar:onejar:lib/server-launcher-24.3.0-19261-main.jar!/config/logback.xml'.
>> stdout jvm 1 | Could not find file `config/logback.xml'. Attempting
>> to load from classpath.
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] Operating
>> System: Linux(4.4.302+)
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] Java
>> Version: 21.0.4
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] GoCD
>> Version: 24.3.0-19261
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] Application
>> arguments: []
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] Working directory
>> : /go-working-dir
>> stdout jvm 1 | [Fri Oct 18 23:01:00 GMT 2024] Starting process:
>> stdout jvm 1 | WrapperManager: Initializing...
>> stdout wrapper | Launching a JVM...
>> stdout wrapper |
>> stdout wrapper | Licensed to ThoughtWorks for GoCD Server
>> stdout wrapper | https://wrapper.tanukisoftware.com
>> stdout "wrapper | Copyright (C) 1999-2024 Tanuki Software, Ltd. All
>> Rights Reserved.
>> stdout ]0;go-server wrapper | Java Service Wrapper Standard Edition
>> 64-bit 3.5.59
>> stdout wrapper | --> Wrapper Started as Console
>> stdout Running go-server...
>> stdout $ exec /usr/local/sbin/tini -g -- /go-working-dir/bin/go-server
>> console
>> stdout $ sed -i -e s@wrapper.logfile
>> =.*@wrapper.logfile=/go-working-dir/logs/go-server-wrapper.log@g -e
>> s...@wrapper.java.command=.*@wrapper.java.command=/gocd-jre/bin/java@g -e
>> s...@wrapper.working.dir=.*@wrapper.working.dir=/go-working-dir@g
>> /go-server/wrapper-config/wrapper.conf
>> stdout /docker-entrypoint.sh: Running custom scripts in
>> /docker-entrypoint.d/ ...
>>
>> On Friday, October 18, 2024 at 5:09:52 PM UTC-6 Lawrence wrote:
>>
>>> It's happening on a fresh install of the docker container, so no data.
>>> Just downloaded the image and started it without mapping any drives
>>>
>>> On Monday, September 23, 2024 at 10:43:10 PM UTC-6 Chad Wilson wrote:
>>>
>>>> Maybe you can share the specific lines it gets stuck on, and something
>>>> of your environment (OS, container runtime version etc)? Doing anything
>>>> unusual with volume mounts or entry points or a custom container image?
>>>>
>>>> If you are upgrading an existing install perhaps there is a lock that's
>>>> stuck on your database?
>>>>
>>>> The containers are all automatically tested that they start up
>>>> correctly within latest Docker on Linux (albeit in a docker-in-docker type
>>>> of environment) and within Kubernetes versions and they also work OK for me
>>>> on MacOS within Colima/Docker.
>>>>
>>>> On Tue, 24 Sept 2024, 04:49 Lawrence, <lawrenc...@gmail.com> wrote:
>>>>
>>>>> Anyone else stuck on upgrading database on a fresh install of the
>>>>> docker container? I'm able to run v24.2.0 container no problem, but
>>>>> v24.3.0
>>>>> didn't complete after an hour.
>>>>>
>>>>> This could be a me problem, but not sure what I could have done since
>>>>> I just downloaded the image and started it.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "go-cd" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to go-cd+un...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/go-cd/caf591ea-eeb8-4745-a875-779860a2bf8an%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/go-cd/caf591ea-eeb8-4745-a875-779860a2bf8an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "go-cd" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to go-cd+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/go-cd/28b822e9-02a0-4138-9d4d-a96054e7c217n%40googlegroups.com
>> <https://groups.google.com/d/msgid/go-cd/28b822e9-02a0-4138-9d4d-a96054e7c217n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to go-cd+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/go-cd/CAA1RwH-MfTs_-SUVXN_yXfaWNfSmWXoOHYvP7XdC8LfUrhC2mQ%40mail.gmail.com.
