-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Andrew David Wong: > A new announcement, "Security challenges for the Qubes build process," > has just been posted on the Qubes website: > > https://www.qubes-os.org/news/2016/05/30/build-security/
Hi, As suggested in this post I implemented a append-only log for qubes-builder. Instead of one log file as the 'qubes.AppendLog' name in the post suggest I decided to create a separate log file per make invocation to keep it a bit simpler. The attached patch consists of three parts: 1) A little wrapper GNUmakefile which calls the logging script. This is a bit ugly but has the advantage that 'make' calls work as before. 2) The logging script logs some information about make invocation and the current state of the git repos and then calls the real qubes-builder Makefile. It is also responsible to copy the output to the remote domain. 3) The rpc service which runs in the log vm. To enable logging set QUBES_BUILD_LOG_CMD in the build vm to 'qrexec-client-vm logs qubesbuilder.BuildLog' (probably in .bash_profile). When you installed the rpc service in the log vm and added a matching policy you now get a log file for every make invocation in ~/QubesIncomingBuildLog/build-vm/. For the official releases I think we should have a git repo where the logs for each published packages is kept (Maybe using a separate key to sign this is a good idea). The attached patch work in principal but might need some fine tuning. a) Do you prefer the GNUmakefile trick or calling the script directly? b) What additional info should be logged? Currently the following is logged: - make arguments - hash of each git HEAD - status and hashes of untracked or modified files except for the top level repo (this includes downloaded sources) - make build-info - make is called with --trace - VERBOSE is set to 2 if unset Do you think more or less should be logged (for example the untracked files are a bit too verbose. logging builder.conf might be a good idea)? HW42 -----BEGIN PGP SIGNATURE----- iQIsBAEBCgAWBQJXspMQDxxodzQyQGlwc3Vtai5kZQAKCRDkrMknimRoFsVZEACb sKtMXH6Mn6HvMdK2T22xwXu/jmYr5Mbn0vXzuETJWL/tRHy2F010m1bVGH6dTZ3+ BcIxo8wJxIMc3iGbDAhB0cEMVg500xh7AjZhY6gfR2gWVaf+YDD2ZNZiJwHieGOH PwEil4UDALzG3jehe6hy9+Np0xg32lpSFMA9UuRqmCspLFtIyyT8m0vXLYFsHAKp UinjU3+/Zcl7RcCVGeTzbE9hNKdx0HxedPwossxlW5mo2kSwK/fYfvk3hrOx6mJo ERKlJpjPW9yu+ZGDZSTcqL0xWxmwFWv8TTcs/VUoBAO/zSwoiCFTy4Nx5pnjpXd5 Wrp95wzKxBovLm0hZm44gO+rFwwDNTSVMhTEKTufYsayB8IPVjQnlizooiuUyBbT 0etkmh1yUMYPCaK5Cdo/EhAm4pqTTq/jxGEQUgxV9E50pmepBhOcnige03kB/q9n +NdY8uVaRy0APUMbvF3CjgXtBU8fs1fPNm0sm1+AkP8SkZYAUO0Y7ylBLTlzYSYG WTkHdMkXbRu6Y34JtdZmtjMxBZacfJIlmw2JNYFQwvK3a4Pd+0STat4empXCrJYl 9tYoqX8YbeRv1gupLosUeg1WSa+Dfbgmf0apEJvezHsEqY2GHszu1DskEbUz26TI 2pcKawObNrE9T38rrgpiul0Pa3LLBxN4te7U5gdD8g== =NjNF -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "qubes-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/491be700-6772-5db6-4b7f-1571df969e9f%40ipsumj.de. For more options, visit https://groups.google.com/d/optout.
From a1f08948ad7a146ac28358cc1858ace9ce74c350 Mon Sep 17 00:00:00 2001 From: HW42 <[email protected]> Date: Tue, 16 Aug 2016 00:04:16 +0200 Subject: [PATCH] add tools for append-only build logs see also https://www.qubes-os.org/news/2016/05/30/build-security/ and #2023 --- GNUmakefile | 9 ++++ rpc-services/qubesbuilder.BuildLog | 87 ++++++++++++++++++++++++++++++++++++++ scripts/log-git-status | 36 ++++++++++++++++ scripts/make-with-log | 37 ++++++++++++++++ 4 files changed, 169 insertions(+) create mode 100644 GNUmakefile create mode 100755 rpc-services/qubesbuilder.BuildLog create mode 100755 scripts/log-git-status create mode 100755 scripts/make-with-log diff --git a/GNUmakefile b/GNUmakefile new file mode 100644 index 0000000..44ca4ca --- /dev/null +++ b/GNUmakefile @@ -0,0 +1,9 @@ +.PHONY: $(MAKECMDGOALS) __default + +.NOTPARALLEL: + +__default $(firstword $(MAKECMDGOALS)): + @scripts/make-with-log $(MAKECMDGOALS) + +$(wordlist 2, $(words $(MAKECMDGOALS)), $(MAKECMDGOALS)): + @# ignore diff --git a/rpc-services/qubesbuilder.BuildLog b/rpc-services/qubesbuilder.BuildLog new file mode 100755 index 0000000..291c8f7 --- /dev/null +++ b/rpc-services/qubesbuilder.BuildLog @@ -0,0 +1,87 @@ +#!/usr/bin/env python + +from __future__ import print_function + +import tempfile +import io +import sys +import os +import errno +import shutil +from datetime import datetime + +def sanitize_line(untrusted_line): + line = bytearray(untrusted_line) + for i, c in enumerate(line): + if c >= 0x20 and c <= 0x7e: + pass + else: + line[i] = 0x2e + return bytearray(line).decode('ascii') + +sys.stdin = io.open(0, 'rb') + +start = datetime.utcnow() + +tmp_log = tempfile.NamedTemporaryFile(prefix="qubes-build-log_", delete=False) + +qrexec_remote = os.getenv('QREXEC_REMOTE_DOMAIN') +if not qrexec_remote: + print('ERROR: QREXEC_REMOTE_DOMAIN not set', file=sys.stderr) + sys.exit(1) + + +def log(msg, remote=True, now=datetime.utcnow()): + if remote: + remote_str = '{}:'.format(qrexec_remote) + else: + remote_str = '>' + + line = '{:%Y-%m-%d %H:%M:%S.%f} {} {}\n'.format(now, remote_str, msg) + + tmp_log.write(line.encode('utf-8')) + +log('starting log', now=start, remote=False) + +while True: + untrusted_line = sys.stdin.readline() + if untrusted_line == b'': + break + + log(sanitize_line(untrusted_line.rstrip(b'\n'))) + + +log('closing log', remote=False) +tmp_log.close() + +file_name_base = ('{home}/QubesIncomingBuildLog/{remote}/' + '{remote}_{time:%Y-%m-%d_%H-%M-%S}').format( + home=os.getenv('HOME', '/'), + remote=qrexec_remote, + time=start) + +try: + os.makedirs(os.path.dirname(file_name_base)) +except OSError as err: + if err.errno != errno.EEXIST: + raise + +try_no = 0 +while True: + if try_no == 0: + file_name = file_name_base + else: + file_name = '{}.{}'.format(file_name_base, try_no) + + try: + fd = os.open(file_name, os.O_CREAT | os.O_EXCL, 0o664) + except OSError as err: + if err.errno == errno.EEXIST: + try_no += 1 + continue + raise + + os.close(fd) + break + +shutil.move(tmp_log.name, file_name) diff --git a/scripts/log-git-status b/scripts/log-git-status new file mode 100755 index 0000000..a5f16ea --- /dev/null +++ b/scripts/log-git-status @@ -0,0 +1,36 @@ +#!/bin/bash + +set -e +set -o pipefail +shopt -s nullglob + +script_path="$(readlink -f "$0")" +base_dir="$(readlink -f "$(dirname "$script_path")/..")" + +repo_abs=. +if [ $# -gt 0 ]; then + repo_abs="$1" +fi +cd "$repo_abs" +repo_abs="$(readlink -f $(git rev-parse --show-toplevel))" + + +repo="${repo_abs#$base_dir}" + +echo ">> qubes-builder$repo $(git rev-parse 'HEAD^{}')" + +ignore_args="" +if [ -n "$repo" ]; then + ignore_args="--ignored" +fi + +git status --porcelain -uall $ignore_args --ignore-submodules=all +git status -z -uall $ignore_args --ignore-submodules=all | sed -zn 'h; /^R/n; g; s/^...//; p' | xargs -r0 sha512sum + +if [ -z "$repo" ]; then + for d in qubes-src/*/; do + $script_path $d + done +else + git submodule -q foreach $script_path +fi diff --git a/scripts/make-with-log b/scripts/make-with-log new file mode 100755 index 0000000..4f9c791 --- /dev/null +++ b/scripts/make-with-log @@ -0,0 +1,37 @@ +#!/bin/bash + +set -e +set -o pipefail + +cd $(dirname "$0")/.. + +# hide wrapper GNUmakefile +if [ -n "$MAKELEVEL" ]; then + (( MAKELEVEL-=1 )) || true +fi + +if [ -z "$QUBES_BUILD_LOG_CMD" ]; then + make -f Makefile "$@" + exit $? +fi + +exec {real_stdout_fd}>&1 + +( +export LC_ALL=C.UTF-8 +echo "> starting build with log" +echo ">> make args: $*" +for v in MAKEFLAGS; do + echo ">> $v=${!v}" +done +$(dirname $0)/log-git-status +make -f Makefile build-info NO_COLOR=1 + +if [ -z "$VERBOSE"]; then + export VERBOSE=2 +fi + +echo "> running make" +make -f Makefile --trace "$@" +echo "> done" +) 2>&1 | tee /dev/fd/$real_stdout_fd | eval "$QUBES_BUILD_LOG_CMD" -- 2.8.1
0001-add-tools-for-append-only-build-logs.patch.sig
Description: PGP signature
