Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package obs-service-go_modules for
openSUSE:Factory checked in at 2022-06-12 17:41:37
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/obs-service-go_modules (Old)
and /work/SRC/openSUSE:Factory/.obs-service-go_modules.new.1548 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "obs-service-go_modules"
Sun Jun 12 17:41:37 2022 rev:4 rq:982163 version:0.5.0
Changes:
--------
---
/work/SRC/openSUSE:Factory/obs-service-go_modules/obs-service-go_modules.changes
2022-05-03 21:19:04.849006889 +0200
+++
/work/SRC/openSUSE:Factory/.obs-service-go_modules.new.1548/obs-service-go_modules.changes
2022-06-12 17:43:21.526503593 +0200
@@ -1,0 +2,13 @@
+Sat Jun 11 01:56:21 UTC 2022 - jkowalc...@suse.com
+
+- Update to version 0.5.0:
+ * README update
+ * Check go mod subcommand return code, log and exit on error
+ * Log go.mod file not found as error not info
+ * Execute go mod subcommands using subprocess.run()
+ * Rework the service to better work with obs_scm
+- Add Require: python3-libarchive-c
+- Drop Require: tar
+- Drop Require: gzip
+
+-------------------------------------------------------------------
Old:
----
obs-service-go_modules-0.4.1.tar.gz
New:
----
obs-service-go_modules-0.5.0.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ obs-service-go_modules.spec ++++++
--- /var/tmp/diff_new_pack.VELZlk/_old 2022-06-12 17:43:21.906504138 +0200
+++ /var/tmp/diff_new_pack.VELZlk/_new 2022-06-12 17:43:21.910504144 +0200
@@ -37,7 +37,7 @@
%define use_test test
%endif
Name: obs-service-%{service}
-Version: 0.4.1
+Version: 0.5.0
Release: 0
Summary: An OBS source service: Download, verify and vendor Go module
dependencies
License: GPL-2.0-or-later
@@ -46,8 +46,7 @@
Source: %{name}-%{version}.tar.gz
BuildRequires: go-md2man
Requires: go >= 1.11
-Requires: gzip
-Requires: tar
+Requires: python3-libarchive-c
BuildArch: noarch
%if %{with needs_external_argparse}
BuildRequires: %{use_python}-argparse
++++++ _service ++++++
--- /var/tmp/diff_new_pack.VELZlk/_old 2022-06-12 17:43:21.942504190 +0200
+++ /var/tmp/diff_new_pack.VELZlk/_new 2022-06-12 17:43:21.946504196 +0200
@@ -3,7 +3,7 @@
<param
name="url">https://github.com/openSUSE/obs-service-go_modules</param>
<param name="scm">git</param>
<param name="exclude">.git</param>
- <param name="revision">v0.4.1</param>
+ <param name="revision">v0.5.0</param>
<param name="versionformat">@PARENT_TAG@</param>
<param name="changesgenerate">enable</param>
<param name="versionrewrite-pattern">v(.*)</param>
++++++ _servicedata ++++++
--- /var/tmp/diff_new_pack.VELZlk/_old 2022-06-12 17:43:21.962504219 +0200
+++ /var/tmp/diff_new_pack.VELZlk/_new 2022-06-12 17:43:21.966504224 +0200
@@ -1,6 +1,6 @@
<servicedata>
<service name="tar_scm">
<param
name="url">https://github.com/openSUSE/obs-service-go_modules</param>
- <param
name="changesrevision">2c5088e247519c74c025cfb6ed16a56dd7b4a5c2</param></service></servicedata>
+ <param
name="changesrevision">3ee3fd053c57ab951c7347243e6f4d017c36c7ee</param></service></servicedata>
(No newline at EOF)
++++++ obs-service-go_modules-0.4.1.tar.gz ->
obs-service-go_modules-0.5.0.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/obs-service-go_modules-0.4.1/README.md
new/obs-service-go_modules-0.5.0/README.md
--- old/obs-service-go_modules-0.4.1/README.md 2022-05-02 10:34:44.000000000
+0200
+++ new/obs-service-go_modules-0.5.0/README.md 2022-06-11 02:15:31.000000000
+0200
@@ -1,5 +1,24 @@
# OBS Source Service `obs-service-go_modules`
+## Contents
+
+- [Overview](#overview)
+- [Usage for packagers](#usage-for-packagers)
+- [Compression format support](#compression-format-support)
+- [OBS Source Service Build Mode
support](#obs-source-service-build-mode-support)
+- [Building Go applications with vendored dependency
modules](#building-go-applications-with-vendored-dependency-modules)
+- [Example](#example)
+- [Example `_service` configuration](#example-_service-configuration)
+- [Transition note](#transition-note)
+- [openSUSE RPM packages built using
`obs-service-go_modules`](#opensuse-rpm-packages-built-using-obs-service-go_modules)
+- [Dependencies](#dependencies)
+- [FAQ](#faq)
+- [Support](#support)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Overview
+
This is the git repository for
[`devel:languages:go/obs-service-go_modules`](https://build.opensuse.org/package/show/devel:languages:go/obs-service-go_modules),
an [Open Build Service (OBS)](https://build.opensuse.org)
@@ -18,21 +37,21 @@
go mod vendor
```
-`obs-service-go_modules` will create a `vendor.tar[.<tar compression>]` archive
-containing the `vendor/` directory populated by `go mod vendor`. The archive
-is generated in the rpm package directory, and can be committed to
+`obs-service-go_modules` will create a `vendor.tar.gz` archive or other
supported compression type
+containing the `vendor/` directory populated by `go mod vendor`.
+The archive is generated in the rpm package directory, and can be committed to
[OBS](https://build.opensuse.org) to facilitate offline Go application package
builds
for [openSUSE](https://www.opensuse.org),
[SUSE](https://www.suse.com), and numerous other distributions.
## Usage for packagers
-Presently it is assumed the Go application is distributed as a tarball named
-`app-0.1.0.tar[.<tar compression>]`, unpacking to `app-0.1.0/`.
-The `<tar compression>` extension can be specified using the `compression`
parameter,
-and defaults to `gz`.
-`obs-service-go_modules` will autodetect tarball archives of the form
`app-0.1.0.tar[.<tar compression>]`,
-where the RPM packaging uses spec file `app.spec`.
+Presently it is assumed the Go application source is distributed as a
compressed tarball named
+`app-0.1.0.tar.gz` or other supported compression type, unpacking to
`app-0.1.0/`.
+The compression type can be specified using the `compression` parameter,
+and defaults to `gz` (gzip).
+`obs-service-go_modules` will autodetect tarball archives of the form
`app-0.1.0.tar.gz`,
+where the RPM packaging uses spec file `app.spec` sharing the base name `app`.
Create a `_service` file containing:
@@ -53,6 +72,65 @@
See [Example](#example) below for typical output with a complete `_service`
file.
+## Compression format support
+
+`obs-service-go_modules` reads and writes compressed tar archives
+using [`libarchive`](https://libarchive.org/) via the Python3 `ctypes` wrapper
+[`python3-libarchive-c`](https://github.com/Changaco/python-libarchive-c).
+While `libarchive` supports numerous compression formats,
+`obs-service-go_modules` usage recommends limiting selections to
+`gz` (default), `xz` and `zstd`.
+Tables of representative compression method relative sizes and timings
+with Go vendored dependency sources (`vendor/`) are shown below.
+
+### Compression sizes with Go dependency sources
+
+| project | version | uncompressed | gz | xz | zstd |
+| ---------- | -------- | ------------ | --- | ---- | ---- |
+| hugo | v0.100.2 | 75M | 11M | 7.5M | 8.3M |
+| kubernetes | v1.24.1 | 129M | 17M | 12M | 14M |
+
+### Compression timing with Go dependency sources
+
+| project | version | gz | xz | zstd |
+| ---------- | -------- | ---- | ----- | ---- |
+| hugo | v0.100.2 | 1.8s | 6.3s | 0.3s |
+| kubernetes | v1.24.1 | 2.9s | 10.9s | 0.4s |
+
+The above are an average of five runs on Intel i7-6820HQ CPU.
+The `zstd` format has a clear advantage in speed with reasonable compression
ratio,
+and is likely to become the default compression method in a future release.
+Decompression timings are closely matched among `gz`, `xz`, and `zstd`
compression methods.
+
+## OBS Source Service Build Mode support
+
+OBS Source Services can run in one of several modes as shown in
+[OBS Documentation: Using Source Services: Modes of
Services](https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.source_service.html#sec.obs.sserv.mode).
+
+Currently the recommended mode is `disabled` (aliased as `manual`) which
implies:
+
+- The service is run locally via explicit CLI call `osc service disabledrun`
+
+- `obs-service-go_modules` and its dependencies
+ `python3`, `libarchive` and `python3-libarchive-c`
+ are installed on the local machine.
+ In particular, `python3-libarchive-c` packages
+ are not available in all SUSE repositories at this time.
+
+- The resulting `vendor.tar.gz` or other supported compression type
+ should be committed and referenced as an RPM `Source:`.
+
+- `Source:` file names are as given in the RPM `.spec`, no `_service:` prefix
is applied.
+
+If and when `obs-service-go_modules` is available on
+[OBS](https://build.opensuse.org),
+additional modes including `Default` will be supported.
+`Default` runs server side after each commit
+and locally before every local build.
+This will enable tighter integration with the
+[tar_scm / obs_scm](https://github.com/openSUSE/obs-service-tar_scm) source
service,
+including uncommitted server-managed `.obscpio` source and vendor archives.
+
## Building Go applications with vendored dependency modules
Go commands support building with vendored dependencies,
@@ -156,7 +234,7 @@
## Transition note
Until such time as `obs-service-go_modules` is available on
-[OBS](https://build.opensuse.org), `vendor.tar[.<tar compression>]` should
+[OBS](https://build.opensuse.org), `vendor.tar.gz` should
be committed along with the Go application release tarball.
## openSUSE RPM packages built using `obs-service-go_modules`
@@ -167,13 +245,27 @@
- [mod](https://build.opensuse.org/package/show/devel:languages:go/mod)
- [mgit](https://build.opensuse.org/package/show/devel:languages:go/mgit)
+## Dependencies
+
+`obs-service-go_modules` requires:
+
+- `python3`
+- `python-libarchive-c` ctypes wrapper for the `libarchive` C library (added
in `v0.5.0`)
+
+The Python standard library supports only gzipped tar archives.
+The `libarchive` dependency was chosen to support additional compression types
+including `xz` and the `cpio_newc` used by `.obscpio` archives.
+Supported compression types are intentionally limited to
+`gz`, `xz`, `zstd` and `cpio_newc` to preserve future flexibility
+in the event eliminating the `python-libarchive-c` dependency is desirable.
+
## FAQ
-### Q: Does `vendor.tar[.<tar compression>]` need to be committed to OBS
package?
+### Q: Does `vendor.tar.gz` need to be committed to OBS package?
A: Currently yes.
As long as `obs-service-go_modules` is run locally via `osc service
disabledrun`,
-then `vendor.tar[.<tar compression>]` should be committed and referenced as an
additional `Source:`.
+then `vendor.tar.gz` should be committed and referenced as an additional
`Source:`.
If and when `obs-service-go_modules` is available on
[OBS](https://build.opensuse.org),
additional strategies should be possible such as a `vendor.cpio`
@@ -207,6 +299,24 @@
as well as provide protections against third-party service outages and
upstream Go modules being removed by the author.
+## Support
+
+`obs-service-go_modules` intends to be compatible with most upstream Go
projects that use best practice source layouts and module conventions.
+If you are packaging a Go application with an uncommon project layout or
nonstandard `go.mod` usage pattern that presents compatibility problems,
+please file an
[issue](https://github.com/openSUSE/obs-service-go_modules/issues) with
description and reference the specific upstream tag or commit.
+While it may not be possible to support every unique upstream layout,
+a maintainer will evaluate feasibility of adding support for that special case
or improve output messages to clearly indicate the error.
+
+## Contributing
+
+In keeping with [support](#support) objectives,
+feature ideas are welcome,
+particularly those that improve idiomatic OBS and RPM usage,
+packaging automation and commonality among Go application package sources.
+It is also a goal to keep manual configuration parameters in `_service` to a
minimum for maintainability.
+Please file proposals as an
[issue](https://github.com/openSUSE/obs-service-go_modules/issues)
+to discuss feasibility and feature design leading to a subsequent
implementation via pull request.
+
## License
GNU General Public License v2.0 or later
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/obs-service-go_modules-0.4.1/go_modules
new/obs-service-go_modules-0.5.0/go_modules
--- old/obs-service-go_modules-0.4.1/go_modules 2022-05-02 10:34:44.000000000
+0200
+++ new/obs-service-go_modules-0.5.0/go_modules 2022-06-11 02:15:31.000000000
+0200
@@ -30,56 +30,73 @@
import logging
import argparse
import re
-import tarfile
+import libarchive
import os
import shutil
from pathlib import Path
-from subprocess import check_output
-from subprocess import CalledProcessError
+from subprocess import run
app_name = "obs-service-go_modules"
description = __doc__
-logging.basicConfig(level=logging.DEBUG)
-log = logging.getLogger(app_name)
-
DEFAULT_COMPRESSION = "gz"
-parser = argparse.ArgumentParser(
- description=description,
formatter_class=argparse.RawDescriptionHelpFormatter
-)
-parser.add_argument("--strategy", default="vendor")
-parser.add_argument("--archive")
-parser.add_argument("--outdir")
-parser.add_argument("--compression", default=DEFAULT_COMPRESSION)
-args = parser.parse_args()
-
-outdir = args.outdir
+def get_archive_parameters(args):
+ archive_format = None
+ archive_compression = None
+ archive_extension = None
+
+ if args.compression == "obscpio" and "cpio" in libarchive.ffi.READ_FORMATS:
+ archive_format = "cpio_newc"
+ archive_compression = None
+ archive_extension = args.compression
+
+ elif args.compression == "tar" and "tar" in libarchive.ffi.READ_FORMATS:
+ archive_format = "gnutar"
+ archive_compression = None
+ archive_extension = args.compression
-def get_archive_extension():
- if args.compression not in tarfile.TarFile.OPEN_METH:
- log.error(f"The specified compression mode is not supported:
\"{args.compression}\"")
- exit(1)
-
- if args.compression == "tar":
- return "tar"
+ else:
+ compression_format = args.compression
+
+ if args.compression == "gz":
+ compression_format = "gzip"
+
+ elif args.compression == "zst":
+ compression_format = "zstd"
- return "tar." + (args.compression)
+ if compression_format not in libarchive.ffi.READ_FILTERS:
+ log.error(
+ f'The specified compression mode is not supported:
"{args.compression}"'
+ )
+ exit(1)
+ archive_format = "gnutar"
+ archive_compression = compression_format
+ archive_extension = "tar." + (args.compression)
-archive_ext = get_archive_extension()
-vendor_tarname = f"vendor.{archive_ext}"
+ return archive_format, archive_compression, archive_extension
+
+
+def basename_from_archive_name(archive_name):
+ return re.sub(
+
"^(?P<service_prefix>_service:[^:]+:)?(?P<basename>.*)\.(?P<extension>obscpio|tar\.[^\.]+)$",
+ r"\g<basename>",
+ archive_name,
+ )
def archive_autodetect():
- """ Find the most likely candidate file that contains go.mod and go.sum.
- For most Go applications this will be app-x.y.z.tar[.<tar
compression>].
- Use the name of the .spec file as the stem for the archive to detect.
- Archive formats supported:
- - .tar.gz
+ """Find the most likely candidate file that contains go.mod and go.sum.
+ For most Go applications this will be app-x.y.z.tar.gz or other supported
compression.
+ Use the name of the .spec file as the stem for the archive to detect.
+ Archive formats supported:
+ - .tar.gz
+ - .tar.xz
+ - .tar.zstd
"""
log.info(f"Autodetecting archive since no archive param provided in
_service")
cwd = Path.cwd()
@@ -89,49 +106,64 @@
log.error(f"Archive autodetection found no spec file under {cwd}")
exit(1)
else:
+ archive = None
spec_dir = spec.parent # typically the same as cwd
spec_stem = spec.stem # stem is app in app.spec
# highest sorted archive under spec_dir
- pattern = f"{spec.stem}*.{archive_ext}"
- archive = next(reversed(sorted(Path(spec_dir).glob(pattern))), None)
+ patterns = [
+ f"{spec.stem}*.tar.*",
+ f"{spec.stem}*.obscpio",
+ f"_service:*:{spec.stem}*tar.*",
+ f"_service:*:{spec.stem}*obscpio",
+ ]
+ for pattern in patterns:
+ log.debug(f"Trying to find archive name with pattern {pattern}")
+ archive = next(reversed(sorted(Path(spec_dir).glob(pattern))),
None)
+
+ if archive:
+ break
+
if not archive:
- log.error(f"Archive autodetection found no matching archive under
{cwd}")
+ log.error(f"Archive autodetection found no matching archive")
exit(1)
- else:
- log.info(f"Archive autodetected at {archive}")
- # Check that app.spec Version: directive value
- # is a substring of detected archive filename
- # Warn if there is disagreement between the versions.
- pattern = re.compile(r"^Version:\s+([\S]+)$", re.IGNORECASE)
- with spec.open(encoding="utf-8") as f:
- for line in f:
- versionmatch = pattern.match(line)
- if versionmatch:
- version = versionmatch.groups(0)[0]
- if not version:
- log.warning(f"Version not found in {spec.name}")
- else:
- if not (version in archive.name):
- log.warning(
- f"Version {version} in {spec.name} does not match
{archive.name}"
- )
- return str(archive.name) # return string not PosixPath
+ log.info(f"Archive autodetected at {archive}")
+ # Check that app.spec Version: directive value
+ # is a substring of detected archive filename
+ # Warn if there is disagreement between the versions.
+ pattern = re.compile(r"^Version:\s+([\S]+)$", re.IGNORECASE)
+ with spec.open(encoding="utf-8") as f:
+ for line in f:
+ versionmatch = pattern.match(line)
+ if versionmatch:
+ version = versionmatch.groups(0)[0]
+ if not version:
+ log.warning(f"Version not found in {spec.name}")
+ else:
+ if not (version in archive.name):
+ log.warning(
+ f"Version {version} in {spec.name} does not match
{archive.name}"
+ )
+ return str(archive.name) # return string not PosixPath
+
+
+def extract(filename, outdir):
+ log.info(f"Extracting {filename} to {outdir}")
-archive = args.archive or archive_autodetect()
-log.info(f"Using archive {archive}")
+ cwd = os.getcwd()
-basename = archive.replace(f".{archive_ext}", "")
+ # make path absolute so we can switch away from the current working
directory
+ filename = os.path.join(cwd, filename)
+ log.info(f"Switching to {outdir}")
+ os.chdir(outdir)
-def extract(filename, dir):
- if filename.endswith(f".{archive_ext}"):
- tar = tarfile.open(filename)
- tar.extractall(path=dir)
- tar.close()
- else:
- log.info(f"Unsupported archive file format for {filename}")
- exit(1)
+ try:
+ libarchive.extract_file(filename)
+ except libarchive.exception.ArchiveError as archive_error:
+ log.error(archive_error)
+
+ os.chdir(cwd)
def find_file(path, filename):
@@ -141,22 +173,39 @@
def cmd_go_mod(cmd, dir):
- try:
- log.info(f"go mod {cmd}")
- output = check_output(["go", "mod", cmd],
cwd=dir).decode("utf-8").strip()
- if output:
- log.info(output)
- except CalledProcessError as e:
- error = e.output.decode("utf-8").strip()
- if error:
- log.info(error)
- raise
+ """Execute go mod subcommand using subprocess.run().
+ Capture both stderr and stdout as text.
+ Log as info or error in this function body.
+ Return CompletedProcess object to caller for control flow.
+ """
+ log.info(f"go mod {cmd}")
+ # subprocess.run() returns CompletedProcess cp
+ cp = run(["go", "mod", cmd], cwd=dir, capture_output=True, text=True)
+ if cp.returncode:
+ log.error(cp.stderr.strip())
+ return cp
def main():
log.info(f"Running OBS Source Service: {app_name}")
- log.info(f"Extracting {archive} to {outdir}")
+ parser = argparse.ArgumentParser(
+ description=description,
formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ parser.add_argument("--strategy", default="vendor")
+ parser.add_argument("--archive")
+ parser.add_argument("--outdir")
+ parser.add_argument("--compression", default=DEFAULT_COMPRESSION)
+ args = parser.parse_args()
+
+ outdir = args.outdir
+
+ archive_format, archive_compression, archive_ext =
get_archive_parameters(args)
+ vendor_tarname = f"vendor.{archive_ext}"
+ archive = args.archive or archive_autodetect()
+ log.info(f"Using archive {archive}")
+
+ basename = basename_from_archive_name(archive)
extract(archive, outdir)
go_mod_path = find_file(outdir, "go.mod")
@@ -164,28 +213,64 @@
go_mod_dir = os.path.dirname(go_mod_path)
log.info(f"Using go.mod found at {go_mod_path}")
else:
- log.info(f"File go.mod not found under {outdir}")
+ log.error(f"File go.mod not found under {outdir}")
exit(1)
if args.strategy == "vendor":
+ # go subcommand sequence:
+ # - go mod download
+ # (is sensitive to invalid module versions, try and log warn if
fails)
+ # - go mod vendor
+ # (also downloads but use separate steps for visibility in OBS
environment)
+ # - go mod verify
+ # (validates checksums)
+
+ # return value cp is type subprocess.CompletedProcess
+ cp = cmd_go_mod("download", go_mod_dir)
+ if cp.returncode:
+ if "invalid version" in cp.stderr:
+ log.warning(
+ f"go mod download is more sensitive to invalid module
versions than go mod vendor"
+ )
+ log.warning(
+ f"if go mod vendor and go mod verify complete, vendoring
is successful"
+ )
+ else:
+ log.error("go mod download failed")
+ exit(1)
- cmd_go_mod("download", go_mod_dir)
- cmd_go_mod("verify", go_mod_dir)
- cmd_go_mod("vendor", go_mod_dir)
+ cp = cmd_go_mod("vendor", go_mod_dir)
+ if cp.returncode:
+ log.error("go mod vendor failed")
+ exit(1)
+
+ cp = cmd_go_mod("verify", go_mod_dir)
+ if cp.returncode:
+ log.error("go mod verify failed")
+ exit(1)
log.info(f"Vendor go.mod dependencies to {vendor_tarname}")
vendor_tarfile = os.path.join(outdir, vendor_tarname)
- vendor_dir = os.path.join(go_mod_dir, "vendor")
- with tarfile.open(vendor_tarfile, "w:" + args.compression) as tar:
- tar.add(vendor_dir, arcname=("vendor"))
+ cwd = os.getcwd()
+ os.chdir(go_mod_dir)
+ vendor_dir = "vendor"
+
+ with libarchive.file_writer(
+ vendor_tarfile, archive_format, archive_compression
+ ) as new_archive:
+ new_archive.add_files(vendor_dir)
+ os.chdir(cwd)
# remove extracted Go application source
try:
to_remove = os.path.join(outdir, basename)
+ log.info(f"Cleaning up working dir {to_remove}")
shutil.rmtree(to_remove)
- except FileNotFoundError as e:
+ except FileNotFoundError:
log.error(f"Could not remove directory not found {to_remove}")
if __name__ == "__main__":
+ logging.basicConfig(level=logging.DEBUG)
+ log = logging.getLogger(app_name)
main()
