Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package openQA for openSUSE:Factory checked
in at 2025-05-20 09:32:19
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/openQA (Old)
and /work/SRC/openSUSE:Factory/.openQA.new.30101 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "openQA"
Tue May 20 09:32:19 2025 rev:692 rq:1277800 version:5.1747282262.9a4e6bb5
Changes:
--------
--- /work/SRC/openSUSE:Factory/openQA/openQA.changes 2025-05-14
17:01:26.675837091 +0200
+++ /work/SRC/openSUSE:Factory/.openQA.new.30101/openQA.changes 2025-05-20
09:34:04.931164181 +0200
@@ -1,0 +2,17 @@
+Thu May 15 14:14:46 UTC 2025 - ok...@suse.com
+
+- Update to version 5.1747282262.9a4e6bb5:
+ * load-templates: with --clean, empty job group YAML templates
+ * Fix phrasing in jobs comment carry over unit test
+ * Bump debug from 4.4.0 to 4.4.1
+ * Bump synckit from 0.11.4 to 0.11.5
+ * Avoid workers getting stuck with old jobs
+ * Avoid duplicate `use Mojo::JSON`
+ * Follow consistent database DUMP_FOLDER target directory
+ * Create new systemd service and timer for database dump and cleanup
+ * dump-templates: dump job groups as they exist, fix group checks
+ * t: load-templates: check harder for what gets loaded
+ * load-templates: job groups: simplify, don't error on group exists
+ * load-templates: fix loading of job templates
+
+-------------------------------------------------------------------
Old:
----
openQA-5.1747157239.98c95eac.obscpio
New:
----
openQA-5.1747282262.9a4e6bb5.obscpio
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ openQA-client-test.spec ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:05.827201720 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:05.831201887 +0200
@@ -18,7 +18,7 @@
%define short_name openQA-client
Name: %{short_name}-test
-Version: 5.1747157239.98c95eac
+Version: 5.1747282262.9a4e6bb5
Release: 0
Summary: Test package for %{short_name}
License: GPL-2.0-or-later
++++++ openQA-devel-test.spec ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:05.855202893 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:05.855202893 +0200
@@ -18,7 +18,7 @@
%define short_name openQA-devel
Name: %{short_name}-test
-Version: 5.1747157239.98c95eac
+Version: 5.1747282262.9a4e6bb5
Release: 0
Summary: Test package for %{short_name}
License: GPL-2.0-or-later
++++++ openQA-test.spec ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:05.883204066 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:05.887204233 +0200
@@ -18,7 +18,7 @@
%define short_name openQA
Name: %{short_name}-test
-Version: 5.1747157239.98c95eac
+Version: 5.1747282262.9a4e6bb5
Release: 0
Summary: Test package for openQA
License: GPL-2.0-or-later
++++++ openQA-worker-test.spec ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:05.919205574 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:05.919205574 +0200
@@ -18,7 +18,7 @@
%define short_name openQA-worker
Name: %{short_name}-test
-Version: 5.1747157239.98c95eac
+Version: 5.1747282262.9a4e6bb5
Release: 0
Summary: Test package for %{short_name}
License: GPL-2.0-or-later
++++++ openQA.spec ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:05.951206914 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:05.955207083 +0200
@@ -20,6 +20,7 @@
%define openqa_extra_services openqa-gru.service openqa-websockets.service
openqa-scheduler.service openqa-enqueue-audit-event-cleanup.service
openqa-enqueue-audit-event-cleanup.timer openqa-enqueue-asset-cleanup.service
openqa-enqueue-git-auto-update.service openqa-enqueue-asset-cleanup.timer
openqa-enqueue-result-cleanup.service openqa-enqueue-result-cleanup.timer
openqa-enqueue-bug-cleanup.service openqa-enqueue-bug-cleanup.timer
openqa-enqueue-git-auto-update.timer openqa-enqueue-needle-ref-cleanup.service
openqa-enqueue-needle-ref-cleanup.timer
%define openqa_services %{openqa_main_service} %{openqa_extra_services}
%define openqa_worker_services openqa-worker.target openqa-slirpvde.service
openqa-vde_switch.service openqa-worker-cacheservice.service
openqa-worker-cacheservice-minion.service
+%define openqa_localdb_services openqa-setup-db.service openqa-dump-db.service
openqa-dump-db.timer
%if %{undefined tmpfiles_create}
%define tmpfiles_create() \
%{_bindir}/systemd-tmpfiles --create %{?*} || : \
@@ -89,7 +90,7 @@
%define devel_requires %devel_no_selenium_requires chromedriver
Name: openQA
-Version: 5.1747157239.98c95eac
+Version: 5.1747282262.9a4e6bb5
Release: 0
Summary: The openQA web-frontend, scheduler and tools
License: GPL-2.0-or-later
@@ -552,13 +553,13 @@
%service_del_postun openqa-continuous-update.timer
%post local-db
-%service_add_post openqa-setup-db.service
+%service_add_post %{openqa_localdb_services}
%preun local-db
-%service_del_preun openqa-setup-db.service
+%service_del_preun %{openqa_localdb_services}
%postun local-db
-%service_del_postun openqa-setup-db.service
+%service_del_postun %{openqa_localdb_services}
%files
%doc README.asciidoc
@@ -797,6 +798,8 @@
%files local-db
%{_unitdir}/openqa-setup-db.service
+%{_unitdir}/openqa-dump-db.service
+%{_unitdir}/openqa-dump-db.timer
%{_unitdir}/openqa-gru.service.requires/postgresql.service
%{_unitdir}/openqa-scheduler.service.requires/postgresql.service
%{_unitdir}/openqa-websockets.service.requires/postgresql.service
@@ -804,6 +807,7 @@
%{_datadir}/openqa/script/dump-db
%{_bindir}/openqa-setup-db
%{_bindir}/openqa-dump-db
+%dir %attr(0755,postgres,root) %{_localstatedir}/lib/openqa/backup
%files single-instance
++++++ node_modules.obscpio ++++++
Binary files old/debug-4.4.0.tgz and new/debug-4.4.0.tgz differ
Binary files old/debug-4.4.1.tgz and new/debug-4.4.1.tgz differ
Binary files old/synckit-0.11.4.tgz and new/synckit-0.11.4.tgz differ
Binary files old/synckit-0.11.5.tgz and new/synckit-0.11.5.tgz differ
++++++ node_modules.spec.inc ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:06.591233729 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:06.595233896 +0200
@@ -80,7 +80,7 @@
Source1079:
https://registry.npmjs.org/dagre-d3/-/dagre-d3-0.6.4.tgz#/dagre-d3-0.6.4.tgz
Source1080:
https://registry.npmjs.org/datatables.net/-/datatables.net-2.3.0.tgz#/datatables.net-2.3.0.tgz
Source1081:
https://registry.npmjs.org/datatables.net-bs5/-/datatables.net-bs5-2.3.0.tgz#/datatables.net-bs5-2.3.0.tgz
-Source1082:
https://registry.npmjs.org/debug/-/debug-4.4.0.tgz#/debug-4.4.0.tgz
+Source1082:
https://registry.npmjs.org/debug/-/debug-4.4.1.tgz#/debug-4.4.1.tgz
Source1083:
https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz#/deep-is-0.1.4.tgz
Source1084:
https://registry.npmjs.org/delaunator/-/delaunator-5.0.1.tgz#/delaunator-5.0.1.tgz
Source1085:
https://registry.npmjs.org/depd/-/depd-2.0.0.tgz#/depd-2.0.0.tgz
@@ -201,7 +201,7 @@
Source1200:
https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz#/statuses-2.0.1.tgz
Source1201:
https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz#/strip-json-comments-3.1.1.tgz
Source1202:
https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz#/supports-color-7.2.0.tgz
-Source1203:
https://registry.npmjs.org/synckit/-/synckit-0.11.4.tgz#/synckit-0.11.4.tgz
+Source1203:
https://registry.npmjs.org/synckit/-/synckit-0.11.5.tgz#/synckit-0.11.5.tgz
Source1204:
https://registry.npmjs.org/timeago/-/timeago-1.6.7.tgz#/timeago-1.6.7.tgz
Source1205:
https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz#/toidentifier-1.0.1.tgz
Source1206:
https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz#/tslib-2.8.1.tgz
++++++ openQA-5.1747157239.98c95eac.obscpio ->
openQA-5.1747282262.9a4e6bb5.obscpio ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/Makefile
new/openQA-5.1747282262.9a4e6bb5/Makefile
--- old/openQA-5.1747157239.98c95eac/Makefile 2025-05-13 19:27:19.000000000
+0200
+++ new/openQA-5.1747282262.9a4e6bb5/Makefile 2025-05-15 06:11:02.000000000
+0200
@@ -71,7 +71,7 @@
install -m 644 -D
--target-directory="$(DESTDIR)/usr/share/openqa/$${f%/*}" "$$f";\
done
- for i in db images testresults pool/1 cache webui/cache; do \
+ for i in db images testresults pool/1 cache webui/cache backup; do \
mkdir -p "$(DESTDIR)"/var/lib/openqa/$$i ;\
done
# shared dirs between openQA web and workers + compatibility links
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/dist/rpm/openQA.spec
new/openQA-5.1747282262.9a4e6bb5/dist/rpm/openQA.spec
--- old/openQA-5.1747157239.98c95eac/dist/rpm/openQA.spec 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/dist/rpm/openQA.spec 2025-05-15
06:11:02.000000000 +0200
@@ -20,6 +20,7 @@
%define openqa_extra_services openqa-gru.service openqa-websockets.service
openqa-scheduler.service openqa-enqueue-audit-event-cleanup.service
openqa-enqueue-audit-event-cleanup.timer openqa-enqueue-asset-cleanup.service
openqa-enqueue-git-auto-update.service openqa-enqueue-asset-cleanup.timer
openqa-enqueue-result-cleanup.service openqa-enqueue-result-cleanup.timer
openqa-enqueue-bug-cleanup.service openqa-enqueue-bug-cleanup.timer
openqa-enqueue-git-auto-update.timer openqa-enqueue-needle-ref-cleanup.service
openqa-enqueue-needle-ref-cleanup.timer
%define openqa_services %{openqa_main_service} %{openqa_extra_services}
%define openqa_worker_services openqa-worker.target openqa-slirpvde.service
openqa-vde_switch.service openqa-worker-cacheservice.service
openqa-worker-cacheservice-minion.service
+%define openqa_localdb_services openqa-setup-db.service openqa-dump-db.service
openqa-dump-db.timer
%if %{undefined tmpfiles_create}
%define tmpfiles_create() \
%{_bindir}/systemd-tmpfiles --create %{?*} || : \
@@ -553,13 +554,13 @@
%service_del_postun openqa-continuous-update.timer
%post local-db
-%service_add_post openqa-setup-db.service
+%service_add_post %{openqa_localdb_services}
%preun local-db
-%service_del_preun openqa-setup-db.service
+%service_del_preun %{openqa_localdb_services}
%postun local-db
-%service_del_postun openqa-setup-db.service
+%service_del_postun %{openqa_localdb_services}
%files
%doc README.asciidoc
@@ -798,6 +799,8 @@
%files local-db
%{_unitdir}/openqa-setup-db.service
+%{_unitdir}/openqa-dump-db.service
+%{_unitdir}/openqa-dump-db.timer
%{_unitdir}/openqa-gru.service.requires/postgresql.service
%{_unitdir}/openqa-scheduler.service.requires/postgresql.service
%{_unitdir}/openqa-websockets.service.requires/postgresql.service
@@ -805,6 +808,7 @@
%{_datadir}/openqa/script/dump-db
%{_bindir}/openqa-setup-db
%{_bindir}/openqa-dump-db
+%dir %attr(0755,postgres,root) %{_localstatedir}/lib/openqa/backup
%files single-instance
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/docs/Installing.asciidoc
new/openQA-5.1747282262.9a4e6bb5/docs/Installing.asciidoc
--- old/openQA-5.1747157239.98c95eac/docs/Installing.asciidoc 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/docs/Installing.asciidoc 2025-05-15
06:11:02.000000000 +0200
@@ -1205,6 +1205,19 @@
Note that jobs marked as incomplete by the stale job detection are not
affected by this
configuration and cloned in any case.
+=== Enable automatic database backup
+[id="automatic_database_cleanup"]
+
+An optional systemd service, `openqa-dump-db.service`, can be enabled to
+perform daily database backups. This service is triggered by the
+`openqa-dump-db.timer`. To enable automatic database backup, run:
+
+[source,sh]
+--------------------------------------------------------------------------------
+systemctl enable --now openqa-dump-db.timer
+--------------------------------------------------------------------------------
+Backups are stored at `/var/lib/openqa/backup`.
+
[id="auditing"]
== Auditing - tracking openQA changes
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/lib/OpenQA/Schema/Result/Jobs.pm
new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/Schema/Result/Jobs.pm
--- old/openQA-5.1747157239.98c95eac/lib/OpenQA/Schema/Result/Jobs.pm
2025-05-13 19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/Schema/Result/Jobs.pm
2025-05-15 06:11:02.000000000 +0200
@@ -4,7 +4,6 @@
package OpenQA::Schema::Result::Jobs;
use Mojo::Base 'DBIx::Class::Core', -signatures;
-use Mojo::JSON 'encode_json';
use Fcntl;
use DateTime;
use OpenQA::Constants qw(WORKER_COMMAND_ABORT WORKER_COMMAND_CANCEL);
@@ -30,7 +29,7 @@
use File::Temp 'tempdir';
use Mojo::Collection;
use Mojo::File qw(tempfile path);
-use Mojo::JSON 'decode_json';
+use Mojo::JSON qw(encode_json decode_json);
use Data::Dump 'dump';
use Text::Diff;
use OpenQA::File;
@@ -1758,23 +1757,29 @@
=cut
sub carry_over_bugrefs ($self) {
- if (my $group = $self->group) { return undef unless
$group->carry_over_bugrefs }
- return undef unless my $prev = $self->_carry_over_candidate;
+ try {
+ if (my $group = $self->group) { return undef unless
$group->carry_over_bugrefs }
+ return undef unless my $prev = $self->_carry_over_candidate;
- my $comments = $prev->comments->search({}, {order_by => {-desc =>
'me.id'}});
- for my $comment ($comments->all) {
- next if !$comment->bugref &&
!exists($comment->text_flags->{carryover});
- my $text = $comment->text;
- my $prev_id = $prev->id;
- $text .= "\n\n(Automatic carryover from t#$prev_id)" if $text !~
qr/Automatic (takeover|carryover)/;
- $text .= "\n(The hook script will not be executed.)"
- if $text !~ qr/The hook script will not be executed/ && defined
$self->hook_script;
- $text .= "\n" unless substr($text, -1, 1) eq "\n";
- my %newone = (text => $text, user_id => $comment->user_id);
- my $comment = $self->comments->create_with_event(\%newone,
{taken_over_from_job_id => $prev_id});
- try { $comment->handle_special_contents }
- catch ($e) { log_info "Unable to evaluate contents of taken-over
comment: $e" }
- return 1;
+ my $comments = $prev->comments->search({}, {order_by => {-desc =>
'me.id'}});
+ for my $comment ($comments->all) {
+ next if !$comment->bugref &&
!exists($comment->text_flags->{carryover});
+ my $text = $comment->text;
+ my $prev_id = $prev->id;
+ $text .= "\n\n(Automatic carryover from t#$prev_id)" if $text !~
qr/Automatic (takeover|carryover)/;
+ $text .= "\n(The hook script will not be executed.)"
+ if $text !~ qr/The hook script will not be executed/ && defined
$self->hook_script;
+ $text .= "\n" unless substr($text, -1, 1) eq "\n";
+ my %newone = (text => $text, user_id => $comment->user_id);
+ my $comment = $self->comments->create_with_event(\%newone,
{taken_over_from_job_id => $prev_id});
+ try { $comment->handle_special_contents }
+ catch ($e) { log_info "Unable to evaluate contents of taken-over
comment: $e" }
+ return 1;
+ }
+ }
+ catch ($e) {
+ my $job_id = $self->id;
+ log_warning "Unable to carry-over bugrefs of job $job_id: $e";
}
return undef;
}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/lib/OpenQA/WebAPI/Controller/API/V1/JobGroup.pm
new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/WebAPI/Controller/API/V1/JobGroup.pm
---
old/openQA-5.1747157239.98c95eac/lib/OpenQA/WebAPI/Controller/API/V1/JobGroup.pm
2025-05-13 19:27:19.000000000 +0200
+++
new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/WebAPI/Controller/API/V1/JobGroup.pm
2025-05-15 06:11:02.000000000 +0200
@@ -3,6 +3,7 @@
package OpenQA::WebAPI::Controller::API::V1::JobGroup;
use Mojo::Base 'Mojolicious::Controller', -signatures;
+use Mojo::JSON;
use Feature::Compat::Try;
use OpenQA::Schema::Result::JobGroups;
@@ -219,7 +220,10 @@
my $check = $self->check_top_level_group;
if ($check != 0) {
return $self->render(
- json => {error => 'Unable to create group with existing name ' .
$validation->param('name')},
+ json => {
+ error => 'Unable to create group with existing name ' .
$validation->param('name'),
+ already_exists => Mojo::JSON->true
+ },
status => 500
);
}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/lib/OpenQA/WebAPI/Controller/API/V1/Worker.pm
new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/WebAPI/Controller/API/V1/Worker.pm
---
old/openQA-5.1747157239.98c95eac/lib/OpenQA/WebAPI/Controller/API/V1/Worker.pm
2025-05-13 19:27:19.000000000 +0200
+++
new/openQA-5.1747282262.9a4e6bb5/lib/OpenQA/WebAPI/Controller/API/V1/Worker.pm
2025-05-15 06:11:02.000000000 +0200
@@ -148,11 +148,11 @@
my $worker = $job->assigned_worker // $job->worker;
my $worker_info = defined $worker ? ('worker ' . $worker->name) : 'worker';
$job->set_property('JOBTOKEN');
- $job->auto_duplicate;
- $job->done(
- result => OpenQA::Jobs::Constants::INCOMPLETE,
- reason => "abandoned: associated $worker_info re-connected but
abandoned the job",
- );
+ try { $job->auto_duplicate }
+ catch ($e) { log_warning("Unable to duplicate job $job_id after being
abandoned by $worker_info: $e") }
+ my $reason = "abandoned: associated $worker_info re-connected but
abandoned the job";
+ try { $job->done(result => INCOMPLETE, reason => $reason) }
+ catch ($e) { log_warning("Unable to incomplete job $job_id after being
abandoned by $worker_info: $e") }
return 1;
}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/package-lock.json
new/openQA-5.1747282262.9a4e6bb5/package-lock.json
--- old/openQA-5.1747157239.98c95eac/package-lock.json 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/package-lock.json 2025-05-15
06:11:02.000000000 +0200
@@ -998,9 +998,9 @@
}
},
"node_modules/debug": {
- "version": "4.4.0",
- "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.0.tgz";,
- "integrity":
"sha512-6WTZ/IxCY/T6BALoZHaE4ctp9xm+Z5kY/pzYaCHRFeyVhojxlrm+46y68HA6hr0TcwEssoxNiDEUJQjfPZ/RYA==",
+ "version": "4.4.1",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.1.tgz";,
+ "integrity":
"sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -2510,13 +2510,13 @@
}
},
"node_modules/synckit": {
- "version": "0.11.4",
- "resolved": "https://registry.npmjs.org/synckit/-/synckit-0.11.4.tgz";,
- "integrity":
"sha512-Q/XQKRaJiLiFIBNN+mndW7S/RHxvwzuZS6ZwmRzUBqJBv/5QIKCEwkBC8GBf8EQJKYnaFs0wOZbKTXBPj8L9oQ==",
+ "version": "0.11.5",
+ "resolved": "https://registry.npmjs.org/synckit/-/synckit-0.11.5.tgz";,
+ "integrity":
"sha512-frqvfWyDA5VPVdrWfH24uM6SI/O8NLpVbIIJxb8t/a3YGsp4AW9CYgSKC0OaSEfexnp7Y1pVh2Y6IHO8ggGDmA==",
"dev": true,
"license": "MIT",
"dependencies": {
- "@pkgr/core": "^0.2.3",
+ "@pkgr/core": "^0.2.4",
"tslib": "^2.8.1"
},
"engines": {
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/script/dump-db
new/openQA-5.1747282262.9a4e6bb5/script/dump-db
--- old/openQA-5.1747157239.98c95eac/script/dump-db 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/script/dump-db 2025-05-15
06:11:02.000000000 +0200
@@ -1,6 +1,6 @@
#!/bin/bash
set -euo pipefail
-DUMP_FOLDER=${DUMP_FOLDER:-"/var/lib/openqa/SQL-DUMPS"}
+DUMP_FOLDER=${DUMP_FOLDER:-"/var/lib/openqa/backup"}
DAYS_TO_KEEP=${DAYS_TO_KEEP:-"7"}
[[ ${1-} = '-h' ]] || [[ ${1-} = '--help' ]] && echo "Dump the openQA
postgresql database to ${DUMP_FOLDER} and clean-up backups older than
${DAYS_TO_KEEP} days." && exit
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/script/openqa-dump-templates
new/openQA-5.1747282262.9a4e6bb5/script/openqa-dump-templates
--- old/openQA-5.1747157239.98c95eac/script/openqa-dump-templates
2025-05-13 19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/script/openqa-dump-templates
2025-05-15 06:11:02.000000000 +0200
@@ -53,6 +53,12 @@
dump as json
+=item B<--convert>
+
+convert legacy non-YAML job groups to modern YAML
+
+dump as json
+
=item B<--help, -h>
print help
@@ -117,7 +123,7 @@
GetOptions(
\%options, "json", "host=s", "apibase=s", "apikey:s", "apisecret:s",
"group=s@", "name=s@", "test=s@", "product=s@", "machine=s@", "full",
- "help|h",
+ "convert", "help|h",
) or usage(1);
usage(0) if $options{help};
@@ -179,24 +185,41 @@
}
if ($tables{'JobGroups'}) {
- my $group = (keys %{$options{group}})[0];
- $url->path($options{apibase} . '/job_templates_scheduling/' . ($group //
''));
- my $res = $client->get($url)->res;
- handle_error($group // 'all groups', $url, $res) unless $res->code &&
$res->code == 200;
- if ($group) {
- # This is already the YAML document of a single group
- push @{$result{JobGroups}}, {group_name => $group, template =>
$res->json};
+ my $group = defined $options{group} ? (keys %{$options{group}})[0] // '' :
'';
+ if ($options{convert}) {
+ # use job_templates_scheduling, which converts if no YAML
+ # string already exists
+ $url->path($options{apibase} . '/job_templates_scheduling/' . $group);
+ my $res = $client->get($url)->res;
+ handle_error($group || 'all groups', $url, $res) unless $res->code &&
$res->code == 200;
+ if ($group) {
+ # This is already the YAML document of a single group
+ push @{$result{JobGroups}}, {group_name => $group, template =>
$res->json};
+ }
+ else {
+ my $yaml = $res->json;
+ foreach my $group (sort keys %$yaml) {
+ push @{$result{JobGroups}}, {group_name => $group, template =>
$yaml->{$group}};
+ }
+ }
}
else {
- my $yaml = $res->json;
- foreach my $group (sort keys %$yaml) {
- push @{$result{JobGroups}}, {group_name => $group, template =>
$yaml->{$group}};
+ # use job_groups and store any existing template strings
+ $url->path($options{apibase} . '/job_groups/');
+ my $res = $client->get($url)->res;
+ handle_error($group || 'all groups', $url, $res) unless $res->code &&
$res->code == 200;
+ my $jgroups = $res->json;
+ for my $jgdata (@$jgroups) {
+ next unless ($jgdata->{template});
+ next if ($group && $jgdata->{name} ne $group);
+ push @{$result{JobGroups}}, {group_name => $jgdata->{name},
template => $jgdata->{template}};
}
}
}
for my $table (qw(Machines TestSuites Products JobTemplates)) {
next unless $tables{$table};
+ next if ($table eq "JobTemplates" && $options{convert});
$url->path($options{apibase} . '/' . decamelize($table));
my $res = $client->get($url)->res;
handle_error($table, $url, $res) unless $res->code && $res->code == 200;
@@ -210,6 +233,8 @@
next if $options{group} && !$options{group}->{$r->{group_name}};
$options{test}->{$r->{test_suite}->{name}} = 1;
$options{machine}->{$r->{machine}->{name}} = 1;
+ # note: $options{product} may not be defined yet, this relies
+ # on perl autovivification
$options{product}->{product_key($r->{product})} = 1;
}
}
@@ -217,7 +242,21 @@
sub handle_row ($table, $r) {
if ($table eq 'JobTemplates') {
return undef if $options{group} && $r->{group_name} &&
!$options{group}->{$r->{group_name}};
- return undef unless $options{product}->{product_key($r->{product})};
+ return undef if $options{product} &&
!$options{product}->{product_key($r->{product})};
+ # skip if we have YAML data for this group. mixing YAML and
+ # non-YAML template data for a group is not a good idea
+ return undef
+ if $r->{group_name} && $result{JobGroups} && grep { $r->{group_name}
eq $_->{group_name} }
+ @{$result{JobGroups}};
+ # see
https://github.com/os-autoinst/openQA/pull/2071#issuecomment-2867855025
+ if ($r->{settings} || $r->{description}) {
+ print STDERR
+ "Settings and/or description unexpectedly present in job
template from legacy (non-YAML) job group "
+ . $r->{group_name} . "!\n";
+ print STDERR "Loading the dumped data would lose the settings.\n";
+ print STDERR "Dump with --convert to convert to YAML format and
preserve settings.\n";
+ exit(1);
+ }
}
return undef if $table eq 'TestSuites' && $options{test} && $r->{name} &&
!$options{test}->{$r->{name}};
return undef if $table eq 'Machines' && $options{machine} && $r->{name} &&
!$options{machine}->{$r->{name}};
@@ -231,8 +270,20 @@
return $r;
}
+sub jt_key ($jt) {
+ my $prod = $jt->{product};
+ return join('',
+ $jt->{group_name},
+ $jt->{test_suite}->{name},
+ $jt->{machine}->{name},
+ $prod->{arch}, $prod->{distri}, $prod->{flavor}, $prod->{version});
+}
+
sub handle_table ($table) {
$result{$table} = [grep { defined } map { handle_row($table, $_) }
@{$result{$table}}];
+ if ($table eq 'JobTemplates') {
+ $result{$table} = [sort { jt_key($a) cmp jt_key($b) }
@{$result{$table}}];
+ }
}
handle_table($_) for keys %result;
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/script/openqa-load-templates
new/openQA-5.1747282262.9a4e6bb5/script/openqa-load-templates
--- old/openQA-5.1747157239.98c95eac/script/openqa-load-templates
2025-05-13 19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/script/openqa-load-templates
2025-05-15 06:11:02.000000000 +0200
@@ -96,6 +96,9 @@
my $url = OpenQA::Client::url_from_host($options{host});
my $client = OpenQA::Client->new(apikey => $options{'apikey'}, apisecret =>
$options{'apisecret'}, api => $url->host);
my @tables = (qw(Machines TestSuites Products JobTemplates JobGroups));
+# clean JobGroups first so we don't hit
+# "must be updated through the YAML template" in the other tables
+my @cleantables = (qw(JobGroups Machines TestSuites Products JobTemplates));
sub print_error ($res) {
if (my $err = $res->error) {
@@ -111,34 +114,33 @@
die "unknown error code - host $url->{host} unreachable?";
}
+sub post_yaml_templates ($group_name, $template) {
+ # Post the job template YAML
+ my $job_templates_url = $url->clone->path($options{apibase} .
'/job_templates_scheduling');
+ return $client->post(
+ $job_templates_url,
+ form => {
+ name => $group_name,
+ template => $template,
+ schema => 'JobTemplates-01.yaml'
+ })->res;
+}
+
sub post_entry ($table, $entry) {
my %param;
if ($table eq 'JobGroups') {
- # Create the group first
+ # Try to create the group first
my $job_groups_url = $url->clone->path($options{apibase} .
'/job_groups');
- my $existing_group_res = $client->get($job_groups_url, form => {name
=> $entry->{group_name}})->res;
- print_error $existing_group_res unless $existing_group_res->is_success;
- my $group_exists = 0;
- for my $group (@{$existing_group_res->json}) {
- next unless $group->{name} eq $entry->{group_name};
- $group_exists = 1
- if $options{update}
- or die "Job group '$entry->{group_name}' already exists. Use
--update to modify existing entries.";
- }
- unless ($group_exists) {
- my $create_res = $client->post($job_groups_url, form => {name =>
$entry->{group_name}})->res;
- print_error($create_res) unless $create_res->is_success;
- }
+ my $create_res = $client->post($job_groups_url, form => {name =>
$entry->{group_name}})->res;
+ # this is what we get from the API if the group exists
+ my $exists = ($create_res->code == 500 &&
$create_res->json->{already_exists});
+ # return 0 (indicating no change) unless --clean or --update passed
+ return 0 if ($exists && !$options{update} && !$options{clean});
+ print_error($create_res) unless ($create_res->is_success || $exists);
# Post the job template YAML
- my $job_templates_url = $url->clone->path($options{apibase} .
'/job_templates_scheduling');
- my $yaml_res = $client->post(
- $job_templates_url,
- form => {
- name => $entry->{group_name},
- template => $entry->{template},
- schema => 'JobTemplates-01.yaml'
- })->res;
+
+ my $yaml_res = post_yaml_templates($entry->{group_name},
$entry->{template});
print_error($yaml_res) unless $yaml_res->is_success;
return 1;
}
@@ -204,27 +206,38 @@
}
}
- my $res = $client->post($table_url, json => \%param)->res;
+ my $type = $table eq 'JobTemplates' ? 'form' : 'json';
+ my $res = $client->post($table_url, $type => \%param)->res;
print_error $res unless $res->is_success;
return 1;
}
if ($options{'clean'}) {
- for my $table (@tables) {
- # we can't clean job groups as they're not deletable unless empty
- next if ($table eq "JobGroups");
+ for my $table (@cleantables) {
my $table_url = $url->clone->path($options{apibase} . '/' .
decamelize($table));
my $res = $client->get($table_url)->res;
if ($res->is_success) {
my $result = $res->json;
- for my $i (0 .. $#{$result->{$table}}) {
- my $id = $result->{$table}->[$i]->{id};
- my $table_url_id = $url->clone->path($options{apibase} . '/' .
decamelize($table) . "/$id");
- $res = $client->delete($table_url_id)->res;
- last unless $res->is_success;
+ # we can't clean job groups as they're not deletable
+ # unless empty, but we *can* replace existing YAML
+ # template strings with 'empty' ones
+ if ($table eq "JobGroups") {
+ # unlike all other tables, is not a hash
+ for my $jg (@$result) {
+ next unless ($jg->{template});
+ $res = post_yaml_templates($jg->{name}, "scenarios:
{}\nproducts: {}\n");
+ last unless $res->is_success;
+ }
+ }
+ else {
+ for my $i (0 .. $#{$result->{$table}}) {
+ my $id = $result->{$table}->[$i]->{id};
+ my $table_url_id = $url->clone->path($options{apibase} .
'/' . decamelize($table) . "/$id");
+ $res = $client->delete($table_url_id)->res;
+ last unless $res->is_success;
+ }
}
}
-
print_error $res unless $res->is_success;
}
}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/systemd/openqa-dump-db.service
new/openQA-5.1747282262.9a4e6bb5/systemd/openqa-dump-db.service
--- old/openQA-5.1747157239.98c95eac/systemd/openqa-dump-db.service
1970-01-01 01:00:00.000000000 +0100
+++ new/openQA-5.1747282262.9a4e6bb5/systemd/openqa-dump-db.service
2025-05-15 06:11:02.000000000 +0200
@@ -0,0 +1,8 @@
+[Unit]
+Description=openQA database dump and cleanup task
+ConditionPathIsReadWrite=/var/lib/openqa/backup
+
+[Service]
+Type=exec
+User=postgres
+ExecStart=/usr/share/openqa/script/dump-db
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/systemd/openqa-dump-db.timer
new/openQA-5.1747282262.9a4e6bb5/systemd/openqa-dump-db.timer
--- old/openQA-5.1747157239.98c95eac/systemd/openqa-dump-db.timer
1970-01-01 01:00:00.000000000 +0100
+++ new/openQA-5.1747282262.9a4e6bb5/systemd/openqa-dump-db.timer
2025-05-15 06:11:02.000000000 +0200
@@ -0,0 +1,9 @@
+[Unit]
+Description=Daily openQA database dump and cleanup task
+
+[Timer]
+OnCalendar=23:40
+Persistent=true
+
+[Install]
+WantedBy=timers.target
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/t/01-compile-check-all.t
new/openQA-5.1747282262.9a4e6bb5/t/01-compile-check-all.t
--- old/openQA-5.1747157239.98c95eac/t/01-compile-check-all.t 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/01-compile-check-all.t 2025-05-15
06:11:02.000000000 +0200
@@ -22,6 +22,7 @@
't/data/openqa/share/tests/opensuse/tests/installation/installer_timezone.pm',
# Skip data files which are supposed to resemble generated output which
has no 'use' statements
't/data/40-templates.pl',
+ 't/data/40-templates-jgs.pl',
't/data/40-templates-more.pl',
't/data/openqa-trigger-from-obs/Proj2::appliances/.api_package',
't/data/openqa-trigger-from-obs/Proj2::appliances/.dirty_status',
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/t/10-jobs.t
new/openQA-5.1747282262.9a4e6bb5/t/10-jobs.t
--- old/openQA-5.1747157239.98c95eac/t/10-jobs.t 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/10-jobs.t 2025-05-15
06:11:02.000000000 +0200
@@ -489,7 +489,7 @@
ok($inv = $job->investigate(git_limit => 23), 'job investigation
ok with test changes');
my $actual_lines = split(/\n/, $inv->{test_log});
my $expected_lines = 7;
- is($actual_lines, $expected_lines, 'test_log have correct number
of lines');
+ is($actual_lines, $expected_lines, 'test_log has the correct
number of lines');
like($inv->{test_log}, qr/^.*file changed/m, 'git log with test
changes');
is $got_limit, 23, 'git_limit was correctly passed';
};
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/t/40-script_load_dump_templates.t
new/openQA-5.1747282262.9a4e6bb5/t/40-script_load_dump_templates.t
--- old/openQA-5.1747157239.98c95eac/t/40-script_load_dump_templates.t
2025-05-13 19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/40-script_load_dump_templates.t
2025-05-15 06:11:02.000000000 +0200
@@ -6,6 +6,7 @@
use FindBin;
use lib "$FindBin::Bin/lib",
"$FindBin::Bin/../external/os-autoinst-common/lib";
use File::Temp qw(tempfile);
+use Mojo::Base -signatures;
use Mojo::File qw(path curfile);
require OpenQA::Test::Database;
use OpenQA::Test::Utils;
@@ -33,6 +34,11 @@
sub decode { Cpanel::JSON::XS->new->relaxed->decode(path(shift)->slurp); }
+sub check_property ($schema, $table, $property, $values) {
+ my @gotprops = sort map { $_->$property } $schema->resultset($table)->all;
+ is_deeply(\@gotprops, $values, "$property entries in $table as expected")
or always_explain \@gotprops;
+}
+
test_once '--help', qr/Usage:/, 'help text shown', 0, 'openqa-load-templates
with no arguments shows usage';
test_once '--host', qr/Option host requires an argument/, 'host argument error
shown', 1, 'required arguments missing';
@@ -58,22 +64,21 @@
$apisecret = 'EXCALIBUR';
my $base_args = "--host $host --apikey $apikey --apisecret $apisecret";
$args = "$base_args $filename";
-my $expected = qr/JobGroups.+=> \{ added => 1, of => 1 \}/;
+my $expected
+ = qr/JobGroups +=> \{ added => 1, of => 1 \},\n +JobTemplates +=> \{ added
=> 0, of => 0 \},\n +Machines +=> \{ added => 1, of => 1 \},\n +Products +=> \{
added => 1, of => 1 \},\n +TestSuites +=> \{ added => 1, of => 1 \}/;
+my $expectednochange
+ = qr/JobGroups +=> \{ added => 0, of => 1 \},\n +JobTemplates +=> \{ added
=> 0, of => 0 \},\n +Machines +=> \{ added => 0, of => 1 \},\n +Products +=> \{
added => 0, of => 1 \},\n +TestSuites +=> \{ added => 0, of => 1 \}/;
test_once $args, $expected, 'Admin may load templates', 0, 'successfully
loaded templates';
-test_once $args, qr/Use --update to modify/, 'Duplicate job group', 255,
'failed on duplicate job group';
+test_once $args, $expectednochange, 'Reload does not modify without --update',
0, 'succeeded without change';
$args = "$base_args --update $filename";
-test_once $args, $expected, 'Update with existing job group', 0, 'updated
template with existing job group';
+test_once $args, $expected, 'Reload with --update modifies', 0, 'updated
template with existing job group';
subtest 'test changing existing entries' => sub {
- # delete job group so that we can load the template again without running
into duplicate job group error
my $t = client(Test::Mojo->new(), apikey => $apikey, apisecret =>
$apisecret);
- $t->get_ok("http://$host/api/v1/job_groups";)->status_is(200);
- my $jobgroup_id = (grep { $_->{name} eq 'openSUSE Leap 42.3 Updates' }
@{$t->tx->res->json})[0]->{id};
-
$t->delete_ok("http://$host/api/v1/job_groups/$jobgroup_id";)->status_is(200);
- $t->get_ok("http://$host/api/v1/test_suites?name=uefi";)->status_is(200);
- my $test_suite_id = $t->tx->res->json->{TestSuites}->[0]->{id};
# overwrite testsuite settings
+ $t->get_ok("http://$host/api/v1/test_suites?name=uefi";)->status_is(200);
+ my $test_suite_id = $t->tx->res->json->{TestSuites}->[0]->{id};
$t->put_ok("http://$host/api/v1/test_suites/$test_suite_id";, json => {name
=> "uefi", settings => {UEFI => '42'}})
->status_is(200);
@@ -91,6 +96,40 @@
'1', 'value changed back during update');
};
+# the test fixtures that we loaded contain:
+# 3 machines '32bit', '64bit', 'Laptop_64'
+# job group 1001 'opensuse' (legacy)
+# job group 1002 'opensuse test' (legacy)
+# 7 test suites textmode, kde, RAID0, client1, server,client2, advanced_kde
+# 3 products, one with 10 job templates (all in 'opensuse' group), two with
none
+# 40-templates.pl which we loaded above contains:
+# 1 machine '128bit'
+# 1 test suite 'uefi' which occurs in no product / job template
+# 1 job group 'openSUSE Leap 42.3 Updates' (modern, empty templates string)
+# 1 product opensuse-42.2-DVD-x86_64
+# 0 job templates
+
+# the 'opensuse' legacy group is unrealistic, as it contains an
+# advanced_kde job template with settings and a description. This is
+# not possible to achieve via the API (only templates produced from
+# a YAML job group can have settings/description). Test that we reject
+# dumping such a group unless --convert is passed:
+$expected = qr,Settings and/or description unexpectedly present.*group
opensuse,;
+dump_templates $base_args, $expected, 'dump_templates fails on legacy group
with settings', 1,
+ 'dump_templates handles error';
+$expected = qr/JobGroups\s*=> \[.*group_name\s*=> "opensuse"/s;
+$args = "$base_args --convert";
+dump_templates $args, $expected, 'dump_templates with --convert', 0,
'dump_templates success with --convert';
+# Also test with --group, for full code coverage
+$args = "$base_args --convert --group opensuse";
+dump_templates $args, $expected, 'dump_templates with --convert --group', 0,
+ 'dump_templates success with --convert --group';
+
+# now wipe the unrealistic settings and description, as following test
+# will correctly fail if they are present
+$schema->resultset('JobTemplateSettings')->delete;
+$schema->resultset('JobTemplates')->update({description => ''});
+
my $fh;
my $tempfilename;
($fh, $tempfilename) = tempfile(UNLINK => 1, SUFFIX => '.json');
@@ -100,39 +139,117 @@
# Clear the data in relevant tables
$schema->resultset($_)->delete for qw(Machines TestSuites Products
JobTemplates JobGroups);
$args = "$base_args $tempfilename";
-$expected = qr/JobGroups.+=> \{ added => 3, of => 3 \}/;
+# we load the modern job group as a JobGroup
+# legacy group 'opensuse' will be loaded implicitly via its JobTemplates
+# legacy group 'opensuse test' disappears at this point as it has no templates
+$expected
+ = qr/JobGroups +=> \{ added => 1, of => 1 \},\n +JobTemplates +=> \{ added
=> 10, of => 10 \},\n +Machines +=> \{ added => 4, of => 4 \},\n +Products +=>
\{ added => 4, of => 4 \},\n +TestSuites +=> \{ added => 8, of => 8 \}/;
test_once $args, $expected, 're-imported fixtures';
my ($rh, $reference) = tempfile(UNLINK => 1, SUFFIX => '.json');
$args = "$base_args --json > $reference";
$expected = qr/^$/;
dump_templates $args, $expected, 're-dumped fixtures';
-is_deeply decode($tempfilename), decode($reference), 'both dumps match';
+eq_or_diff_text decode($tempfilename), decode($reference), 'both dumps match';
+# check we have at least vaguely the stuff we intend to have
+check_property($schema, 'Machines', 'name', ['128bit', '32bit', '64bit',
'Laptop_64']);
+check_property($schema, 'JobGroups', 'name', ['openSUSE Leap 42.3 Updates',
'opensuse']);
+check_property($schema, 'JobTemplates', 'prio', [40, 40, 40, 40, 40, 40, 40,
40, 40, 40]);
+check_property($schema, 'Products', 'arch', ['i586', 'ppc64', 'x86_64',
'x86_64']);
+check_property($schema, 'TestSuites', 'name',
+ ['RAID0', 'advanced_kde', 'client1', 'client2', 'kde', 'server',
'textmode', 'uefi']);
+my $modjg = $schema->resultset('JobGroups')->find({name => 'openSUSE Leap 42.3
Updates'});
+ok $modjg->template, 'modern group template reloaded';
+my $legjg = $schema->resultset('JobGroups')->find({name => 'opensuse'});
+ok !$legjg->template, 'legacy group has no template';
subtest 'dump_templates tests' => sub {
$args = $base_args;
dump_templates "$args Products42", qr/Invalid table.*42/, 'Error on
non-existant table', 1, 'table error';
- $args .= " --test uefi --machine 32bit --group opensuse --product bar
--full JobTemplates";
+ $args .= " --test uefi --machine 128bit --group opensuse --product bar
--full JobTemplates";
$expected = qr/JobTemplates\s*=> \[.*group_name\s*=> "opensuse"/s;
dump_templates $args, $expected, 'dump_templates with options', 0,
'dump_templates success with options';
- $args = "$base_args --test uefi --machine 32bit --group \"openSUSE Leap
42\" --product bar --full JobTemplates";
+ # this test intends to hit job_templates_scheduling/openSUSE Leap 42
+ # and find it doesn't exist; we need --convert to use that endpoint
+ $args = "$base_args --convert --group \"openSUSE Leap 42\"";
$expected = qr/ERROR requesting.*404 - Not Found/;
dump_templates $args, $expected, 'dump_templates fails on wrong group', 1,
'dump_templates handles error';
};
+# now dump with --convert and reload, which will convert 'opensuse' to a YAML
group
+$args = "$base_args --convert --json > $tempfilename";
+$expected = qr/^$/;
+dump_templates $args, $expected, 'dumped fixtures';
+# clear data again
+$schema->resultset($_)->delete for qw(Machines TestSuites Products
JobTemplates JobGroups);
+$args = "$base_args $tempfilename";
+$expected
+ = qr/JobGroups +=> \{ added => 2, of => 2 \},\n +Machines +=> \{ added => 4,
of => 4 \},\n +Products +=> \{ added => 4, of => 4 \},\n +TestSuites +=> \{
added => 8, of => 8 \}/;
+test_once $args, $expected, 're-imported fixtures';
+$legjg = $schema->resultset('JobGroups')->find({name => 'opensuse'});
+ok $legjg->template, 'formerly-legacy group now has template';
+# the right number of job templates showed up, with priorities
+check_property($schema, 'JobTemplates', 'prio', [40, 40, 40, 40, 40, 40, 40,
40, 40, 40]);
+
# Clear the data in relevant tables again
$schema->resultset($_)->delete for qw(Machines TestSuites Products
JobTemplates JobGroups);
# load the templates file with 2 machines
$args = "--host $host --apikey $apikey --apisecret $apisecret $morefilename";
$expected = qr/Machines.+=> \{ added => 2, of => 2 \}/;
test_once $args, $expected, 'imported MOAR fixtures';
-# wipe jobgroups manually as --clean explicitly skips it
-$schema->resultset("JobGroups")->delete;
# now load the templates file with only 1 machine, with --clean
$args = "--host $host --apikey $apikey --apisecret $apisecret --clean
$filename";
$expected = qr/Machines.+=> \{ added => 1, of => 1 \}/;
test_once $args, $expected, 'imported original fixtures';
is $schema->resultset('Machines')->count, 1, "only one machine is loaded";
my $machine = $schema->resultset('Machines')->first;
-is $machine->name, "32bit", "correct machine is loaded";
+is $machine->name, "128bit", "correct machine is loaded";
+
+# Clear the data in relevant tables again
+$schema->resultset($_)->delete for qw(Machines TestSuites Products
JobTemplates JobGroups);
+# load a template file with YAML job group settings and description
+$args = "$base_args t/data/40-templates-jgs.pl";
+$expected = qr/JobGroups +=> \{ added => 1, of => 1 \}/;
+test_once $args, $expected, 'imported YAML job groups';
+# check we got the expected job template with settings and description
+my $found = 0;
+for my $jt ($schema->resultset('JobTemplates')->all) {
+ next unless $jt->test_suite->name eq 'advanced_kde';
+ $found++;
+ $expected = {DESKTOP => 'advanced_kde', ADVANCED => '1'};
+ eq_or_diff $jt->settings_hash, $expected, 'advanced_kde job template has
expected settings';
+ is $jt->description, 'such advanced very test', 'job template has expected
description';
+}
+is $found, 1, 'exactly one advanced_kde job template was found';
+
+# now let's test --update with YAML groups
+# clear the templates, but leave the job group in existence
+$schema->resultset($_)->delete for qw(Machines TestSuites Products
JobTemplates);
+$schema->resultset('JobGroups')->update({template => ''});
+# test reload without --clean or --update does nothing as the group exists
+$expected = qr/JobGroups +=> \{ added => 0, of => 1 \}/;
+test_once $args, $expected, 're-import without --clean or --update does
nothing';
+# test with --update and 'empty' state
+$args = "$base_args --update t/data/40-templates-jgs.pl";
+$expected = qr/JobGroups +=> \{ added => 1, of => 1 \}/;
+test_once $args, $expected, 're-import with --update works';
+is $schema->resultset('JobTemplates')->count, 3, 'job templates were loaded';
+
+# to test YAML --clean, let's rename the group from 40-templates-jgs
+# to 'fedora' and add a legacy group
+$schema->resultset('JobGroups')->update({name => 'fedora'});
+$schema->resultset('JobGroups')->create({name => 'legacy', description =>
'legacy group'});
+$args = "$base_args --clean t/data/40-templates-jgs.pl";
+$expected = qr/JobGroups +=> \{ added => 1, of => 1 \}/;
+test_once $args, $expected, 're-import with --clean works';
+is $schema->resultset('JobTemplates')->count, 3, 'job templates were loaded';
+is $schema->resultset('JobGroups')->count, 3, 'we now have three job groups';
+my $fedora = $schema->resultset('JobGroups')->find({name => 'fedora'});
+my $opensuse = $schema->resultset('JobGroups')->find({name => 'opensuse'});
+my $legacy = $schema->resultset('JobGroups')->find({name => 'legacy'});
+is $fedora->template, "scenarios: {}\nproducts: {}\n", 'fedora group template
emptied';
+is $legacy->template, undef, 'legacy group template still undefined';
+ok $opensuse->template, 'opensuse group has template';
+# make sure all job templates are for opensuse group not fedora group
+check_property($schema, 'JobTemplates', 'group_id', [$opensuse->id,
$opensuse->id, $opensuse->id]);
done_testing;
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/t/api/01-workers.t
new/openQA-5.1747282262.9a4e6bb5/t/api/01-workers.t
--- old/openQA-5.1747157239.98c95eac/t/api/01-workers.t 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/api/01-workers.t 2025-05-15
06:11:02.000000000 +0200
@@ -6,8 +6,10 @@
use FindBin;
use lib "$FindBin::Bin/../lib",
"$FindBin::Bin/../../external/os-autoinst-common/lib";
+use Mojo::Base -signatures;
use Test::Mojo;
use Test::Warnings qw(:all :report_warnings);
+use Test::Output qw(combined_like);
use Mojo::URL;
use OpenQA::Test::TimeLimit '10';
use OpenQA::Test::Case;
@@ -146,17 +148,65 @@
is($jobs->find($running_job_id)->state, RUNNING, 'assigned job still
running');
};
- subtest 'previous job incompleted when worker doing something else' => sub
{
- delete $registration_params{job_id};
- $t->post_ok('/api/v1/workers', form => \%registration_params)
- ->status_is(200, 'register existing worker passing no job ID')
- ->json_is('/id' => $worker_id, 'worker ID returned');
- return always_explain $t->tx->res->json unless $t->success;
+ my $expected_breakage = '';
+ my $expected_warning = qr//;
+ my $test_registration = sub {
+ $schema->txn_begin;
+ combined_like sub { $t->post_ok('/api/v1/workers', form =>
\%registration_params) },
+ $expected_warning, 'expected warning logged';
+ $t->status_is(200, 'register existing worker passing no job ID');
+ $t->json_is('/id' => $worker_id, 'worker ID returned');
+ always_explain $t->tx->res->json unless $t->success;
+ return $schema->txn_rollback if $expected_breakage eq 'worker';
+ is $workers->find($worker_id)->job_id, undef, 'worker has no longer an
assigned job';
+ return $schema->txn_rollback if $expected_breakage eq 'job';
my $incomplete_job = $jobs->find($running_job_id);
- is($incomplete_job->state, DONE, 'assigned job set to done');
- ok($incomplete_job->result eq INCOMPLETE || $incomplete_job->result eq
PARALLEL_RESTARTED,
- 'assigned job considered incomplete or parallel restarted')
+ is $incomplete_job->state, DONE, 'assigned job set to done';
+ ok $incomplete_job->result eq INCOMPLETE || $incomplete_job->result eq
PARALLEL_RESTARTED,
+ 'assigned job considered incomplete or parallel restarted'
or always_explain 'actual job result: ' . $incomplete_job->result;
+ $schema->txn_rollback;
+ };
+
+ subtest 'previous job incompleted when worker doing something else' => sub
{
+ delete $registration_params{job_id};
+
+ subtest 'no errors occurred' => $test_registration;
+
+ # test the worst case where workers will end up stuck with a job
+ # note: This is not supposed to happen in production as errors are now
supposed to be caught
+ # earlier by the cases tested in further subtests. However, this
warning turned out to
+ # be useful when investigating problems so we should keep it and
have this test.
+ my $mock =
Test::MockModule->new('OpenQA::WebAPI::Controller::API::V1::Worker');
+ $mock->redefine(_incomplete_previous_job => sub { die 'foo1' });
+ $expected_breakage = 'worker';
+ $expected_warning = qr/Unable to incomplete.*reschedule.*abandoned by
worker 2: foo1/;
+ subtest 'incompleting previous job fails' => $test_registration;
+
+ # test that everything else works as expected despite failing
duplication
+ $mock = Test::MockModule->new('OpenQA::Schema::Result::Jobs');
+ $mock->redefine(auto_duplicate => sub { die 'foo2' });
+ $expected_breakage = '';
+ $expected_warning = qr/Unable to duplicate job.*abandoned by worker
remotehost:1: foo2/;
+ subtest 'failure when duplicating job does not prevent anything else'
=> $test_registration;
+
+ # test that when we cannot set the job to done the worker is at least
freed from its job
+ # note: It is very important to free the worker from its job as it
would otherwise be stuck
+ # forever. The jobs seem to be mostly cancelled by obsoletion in
practices anyway and
+ # can also be cancelled manually by users if needed.
+ $mock->unmock('auto_duplicate');
+ $mock->redefine(done => sub { die 'foo3' });
+ $expected_breakage = 'job';
+ $expected_warning = qr/Unable to incomplete job.*abandoned by worker
remotehost:1: foo3/;
+ subtest 'worker still freed from its job, even if job cannot be set
done' => $test_registration;
+
+ # test that failures when reading results (e.g. for carry over) are
not preventing anything else to happen
+ $mock = Test::MockModule->new('OpenQA::Schema::Result::JobModules');
+ $jobs->find($running_job_id)->modules->create({name => 'foo', script
=> 'bar', category => 'testsuite'});
+ $mock->redefine(results => sub ($self, %options) { die 'foo4' });
+ $expected_breakage = '';
+ $expected_warning = qr/Unable to carry-over bugrefs of job.*: foo4/;
+ subtest 'failure when reading job module results does not prevent
anything else' => $test_registration;
};
};
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/t/data/40-templates-jgs.pl
new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates-jgs.pl
--- old/openQA-5.1747157239.98c95eac/t/data/40-templates-jgs.pl 1970-01-01
01:00:00.000000000 +0100
+++ new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates-jgs.pl 2025-05-15
06:11:02.000000000 +0200
@@ -0,0 +1,46 @@
+{
+ JobGroups => [
+ {
+ group_name => "opensuse",
+ template =>
+"defaults:\n i586:\n machine: 64bit\n priority: 50\nproducts:\n
opensuse-13.1-DVD-i586:\n distri: opensuse\n flavor: DVD\n version:
\'13.1\'\nscenarios:\n i586:\n opensuse-13.1-DVD-i586:\n - textmode:\n
machine: 32bit\n priority: 40\n - textmode:\n machine:
64bit\n priority: 40\n - advanced_kde:\n description: such
advanced very test\n priority: 40\n settings:\n
ADVANCED: \'1\'\n DESKTOP: advanced_kde",
+ },
+ ],
+ JobTemplates => [],
+ Machines => [
+ {
+ backend => "qemu",
+ name => "32bit",
+ settings => [],
+ },
+ {
+ backend => "qemu",
+ name => "64bit",
+ settings => [],
+ },
+ ],
+ Products => [
+ {
+ arch => "i586",
+ distri => "opensuse",
+ flavor => "DVD",
+ settings => [],
+ version => 13.1,
+ },
+ ],
+ TestSuites => [
+ {
+ name => "textmode",
+ settings => [{key => "DESKTOP", value => "textmode"}, {key =>
"VIDEOMODE", value => "text"},],
+ },
+ {
+ name => "advanced_kde",
+ description => 'See kde for simple test',
+ settings => [
+ {key => "DESKTOP", value => "kde"},
+ {key => "START_AFTER_TEST", value => "kde,textmode"},
+ {key => "PUBLISH_HDD_1", value =>
'%DISTRI%-%VERSION%-%ARCH%-%DESKTOP%-%QEMUCPU%.qcow2'}
+ ],
+ },
+ ],
+}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/t/data/40-templates-more.pl
new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates-more.pl
--- old/openQA-5.1747157239.98c95eac/t/data/40-templates-more.pl
2025-05-13 19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates-more.pl
2025-05-15 06:11:02.000000000 +0200
@@ -9,12 +9,12 @@
Machines => [
{
backend => "qemu",
- name => "32bit",
+ name => "128bit",
settings => [],
},
{
backend => "qemu",
- name => "64bit",
+ name => "256bit",
settings => [],
},
],
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore'
old/openQA-5.1747157239.98c95eac/t/data/40-templates.json
new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates.json
--- old/openQA-5.1747157239.98c95eac/t/data/40-templates.json 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates.json 1970-01-01
01:00:00.000000000 +0100
@@ -1,35 +0,0 @@
-{
- "Machines" : [
- {
- "name" : "32bit",
- "backend" : "qemu"
- }
- ],
- "JobGroups" : [
- {
- "group_name" : "openSUSE Leap 42.3 Updates"
- }
- ],
- "Products" : [
- {
- "arch" : "x86_64",
- "version" : "42.2",
- "flavor" : "DVD",
- "distri" : "opensuse"
- }
- ],
- "JobTemplates" : [
- ],
- "TestSuites" : [
- {
- "description" : "Maintainer: averagea\n\nThat's some test right
there",
- "name" : "uefi",
- "settings" : [
- {
- "key" : "DESKTOP",
- "value" : "gnome"
- }
- ]
- }
- ]
-}
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/openQA-5.1747157239.98c95eac/t/data/40-templates.pl
new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates.pl
--- old/openQA-5.1747157239.98c95eac/t/data/40-templates.pl 2025-05-13
19:27:19.000000000 +0200
+++ new/openQA-5.1747282262.9a4e6bb5/t/data/40-templates.pl 2025-05-15
06:11:02.000000000 +0200
@@ -9,7 +9,7 @@
Machines => [
{
backend => "qemu",
- name => "32bit",
+ name => "128bit",
settings => [],
},
],
++++++ openQA.obsinfo ++++++
--- /var/tmp/diff_new_pack.SYkaFg/_old 2025-05-20 09:34:26.848082377 +0200
+++ /var/tmp/diff_new_pack.SYkaFg/_new 2025-05-20 09:34:26.852082545 +0200
@@ -1,5 +1,5 @@
name: openQA
-version: 5.1747157239.98c95eac
-mtime: 1747157239
-commit: 98c95eac234957bfee6d467b3e9f630085a83dde
+version: 5.1747282262.9a4e6bb5
+mtime: 1747282262
+commit: 9a4e6bb524ff38106d457d394d5492340a32e92f
