On Fri, Dec 24, 2021 at 1:06 PM Marcin Sobczyk <msobc...@redhat.com> wrote:
>
> Hi All,
>
> OST currently fails all the time during engine setup.
> Here's a piece of ansible log that's seen repeatedly and I think
> describes the problem:
>
> 11:07:54 E "engine-config",
> 11:07:54 E "-s",
> 11:07:54 E "OvfUpdateIntervalInMinutes=10"
> 11:07:54 E ],
> 11:07:54 E "delta": "0:00:01.142926",
> 11:07:54 E "end": "2021-12-24 11:06:37.894810",
> 11:07:54 E "invocation": {
> 11:07:54 E "module_args": {
> 11:07:54 E "_raw_params": "engine-config -s
> OvfUpdateIntervalInMinutes='10' ",
> 11:07:54 E "_uses_shell": false,
> 11:07:54 E "argv": null,
> 11:07:54 E "chdir": null,
> 11:07:54 E "creates": null,
> 11:07:54 E "executable": null,
> 11:07:54 E "removes": null,
> 11:07:54 E "stdin": null,
> 11:07:54 E "stdin_add_newline": true,
> 11:07:54 E "strip_empty_ends": true,
> 11:07:54 E "warn": false
> 11:07:54 E }
> 11:07:54 E },
> 11:07:54 E "item": {
> 11:07:54 E "key": "OvfUpdateIntervalInMinutes",
> 11:07:54 E "value": "10"
> 11:07:54 E },
> 11:07:54 E "msg": "non-zero return code",
> 11:07:54 E "rc": 1,
> 11:07:54 E "start": "2021-12-24 11:06:36.751884",
> 11:07:54 E "stderr": "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false",
> 11:07:54 E "stderr_lines": [
> 11:07:54 E "Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false"
> 11:07:54 E ],
> 11:07:54 E "stdout": "Error loading module from
> /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml",
> 11:07:54 E "stdout_lines": [
> 11:07:54 E "Error loading module from
> /usr/share/ovirt-engine/modules/common/org/springframework/main/module.xml"
>
> We do set some config values for the engine in OST when running
> engine-setup. I tried commenting these out, but then engine failed
> health check anyway:
>
> "Status code was 503 and not [200]: HTTP Error 503: Service Unavailable"
>
> The last working set of OST images was the one from Dec 23, 2021 2:05:08
> AM. The first broken one is from Dec 24, 2021 2:05:09 AM. The shipped
> ovirt-engine RPMs doesn't seem to contain any important changes for
> these two sets, but AFAICS the newer ovirt-dependencies RPM did take in
> a couple of patches that look suspicious [1][2][3]. The patches were
> merged on November 16th, but it seems they were first used in that
> broken set from Dec 24 (the one from Dec 23 seems to contain
> ovirt-dependencies RPM based on this [4] commit).
>
> I wanted to try out an older version of ovirt-dependencies, but I think
> they were wiped out from resources.ovirt.org.
>
> I will disable cyclic el8stream OST runs for now, cause all of them
> fail. If there's anyone working and able to make a build with those
> patches reverted and test that out, please ping me and I'll re-enable them.
>
> Regards, Marcin
>
> [1] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114699
> [2] https://gerrit.ovirt.org/c/ovirt-dependencies/+/113877
This is the one that broke us - it replaced a set of shell scripts
with maven. These scripts also created:
# ls -l /usr/share/java/ovirt-dependencies/4.4
total 0
lrwxrwxrwx. 1 root root 24 Nov 4 18:00 gwt-servlet.jar ->
../gwt-servlet-2.9.0.jar
lrwxrwxrwx. 1 root root 31 Nov 4 18:00 spring-aop.jar ->
../spring-aop-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 33 Nov 4 18:00 spring-beans.jar ->
../spring-beans-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 35 Nov 4 18:00 spring-context.jar ->
../spring-context-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-core.jar ->
../spring-core-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-expression.jar ->
../spring-expression-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 38 Nov 4 18:00 spring-instrument.jar ->
../spring-instrument-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-jdbc.jar ->
../spring-jdbc-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 32 Nov 4 18:00 spring-test.jar ->
../spring-test-5.0.4.RELEASE.jar
lrwxrwxrwx. 1 root root 30 Nov 4 18:00 spring-tx.jar ->
../spring-tx-5.0.4.RELEASE.jar
But now there is no '4.4' directory. The engine spec still has:
%global ovirt_dependencies ovirt-dependencies/4.4
despite being at 4.5 and despite also ovirt-dependencies being
bumped to 4.5 a few days ago.
Not sure what the exact plans are regarding this change - if it's
just a bug, I'll let Martin/Artur fix (by perhaps adding a 4.5
directory with links and point the 4.5 engine there, or make it
provide both - not sure what the plans are). I did not try to
revert, anyway, because quite many other related patches were
merged since, including the move to github/copr.
For now, I pushed this simple workaround to OST [1]:
# ln -s . /usr/share/java/ovirt-dependencies/4.4
# ls -l /usr/share/java/ovirt-dependencies/
total 4724
lrwxrwxrwx. 1 root root 1 Dec 27 08:28 4.4 -> .
-rw-r--r--. 1 root root 360096 Dec 26 23:39 spring-aop.jar
-rw-r--r--. 1 root root 654084 Dec 26 23:39 spring-beans.jar
-rw-r--r--. 1 root root 1079129 Dec 26 23:39 spring-context.jar
-rw-r--r--. 1 root root 1216476 Dec 26 23:39 spring-core.jar
-rw-r--r--. 1 root root 263472 Dec 26 23:39 spring-expression.jar
-rw-r--r--. 1 root root 1452 Dec 26 23:39 spring-instrument.jar
-rw-r--r--. 1 root root 15898 Dec 26 23:39 spring-jcl.jar
-rw-r--r--. 1 root root 391565 Dec 26 23:39 spring-jdbc.jar
-rw-r--r--. 1 root root
590394 Dec 26 23:39 spring-test.jar
-rw-r--r--. 1 root root 249289 Dec 26 23:39 spring-tx.jar
With this:
# ls -l /usr/share/ovirt-engine/modules/common/org/springframework/main/
total 4
-rw-r--r--. 1 root root 737 Dec 23 15:29 module.xml
lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-aop.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-aop.jar
lrwxrwxrwx. 1 root root 53 Dec 23 15:42 spring-asm.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-asm.jar
lrwxrwxrwx. 1 root root 55 Dec 23 15:42 spring-beans.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-beans.jar
lrwxrwxrwx. 1 root
root 57 Dec 23 15:42 spring-context.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-context.jar
lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-core.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-core.jar
lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-expression.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-expression.jar
lrwxrwxrwx. 1 root root 60 Dec 23 15:42 spring-instrument.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-instrument.jar
lrwxrwxrwx. 1 root root 54 Dec 23 15:42 spring-jdbc.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-jdbc.jar
lrwxrwxrwx. 1 root root 52 Dec 23 15:42 spring-tx.jar ->
/usr/share/java/ovirt-dependencies/4.4/spring-tx.jar
This unbreaks all the links despite one: spring-asm.jar is still missing.
No idea why. It's missing also on my slightly older setup from
ovirt-dependencies commit ca19602 (and engine 3326bfea06). I guess it might
be unrelated.
I am testing [1] now locally. It already passed test_add_cpu_profile
so seems promising.
Best regards,
[1] https://gerrit.ovirt.org/c/ovirt-system-tests/+/118150
> [3] https://gerrit.ovirt.org/c/ovirt-dependencies/+/114654
> [4] https://gerrit.ovirt.org/c/ovirt-dependencies/+/117459
> _______________________________________________
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MON3B5NSSXBL2BFSLYYIMOTKUR2CK732/
--
Didi
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Z5ATDX5D7M4E35J5Z7DJI4O36VKEM75L/
