The branch, master has been updated via dc6edc48818 WHATSNEW.txt: Improved winbind logging and samba-log-parser via 15fdf7b36f3 docs-xml:manpages: Add man page for samba-log-parser via c9fa3dff8ca s3:script: Add samba-log-parser from fcedf5514b1 smbcacls/smbcquotas: check for valid UNC path
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit dc6edc488189cf02c8e75016104324d497273152 Author: Pavel Filipenský <pfilipen...@samba.org> Date: Wed Jun 7 14:55:49 2023 +0200 WHATSNEW.txt: Improved winbind logging and samba-log-parser Signed-off-by: Pavel Filipenský <pfilipen...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> Autobuild-User(master): Andreas Schneider <a...@cryptomilk.org> Autobuild-Date(master): Wed Jun 7 15:06:07 UTC 2023 on atb-devel-224 commit 15fdf7b36f3c63be70483d72af3b46b29d4034b4 Author: Pavel Filipenský <pfilipen...@samba.org> Date: Tue May 9 14:09:55 2023 +0200 docs-xml:manpages: Add man page for samba-log-parser Signed-off-by: Pavel Filipenský <pfilipen...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> commit c9fa3dff8ca38d27b6452c6b854e45ec44de4932 Author: Pavel Filipenský <pfilipen...@samba.org> Date: Thu Feb 9 16:48:49 2023 +0100 s3:script: Add samba-log-parser Signed-off-by: Pavel Filipenský <pfilipen...@samba.org> Signed-off-by: Andreas Schneider <a...@samba.org> Pair-Programmed-With: Andreas Schneider <a...@samba.org> ----------------------------------------------------------------------- Summary of changes: WHATSNEW.txt | 10 + docs-xml/manpages/samba-log-parser.1.xml | 147 ++++++++++++++ docs-xml/wscript_build | 1 + source3/script/samba-log-parser | 325 +++++++++++++++++++++++++++++++ source3/script/wscript_build | 1 + 5 files changed, 484 insertions(+) create mode 100644 docs-xml/manpages/samba-log-parser.1.xml create mode 100755 source3/script/samba-log-parser Changeset truncated at 500 lines: diff --git a/WHATSNEW.txt b/WHATSNEW.txt index 2b472aa0cdc..8fbf1b59dbd 100644 --- a/WHATSNEW.txt +++ b/WHATSNEW.txt @@ -36,6 +36,15 @@ an implementation written in python. The new function can be imported via `import samba.gp`. The python implementation connects to Active Directory using the SamDB module, instead of ADS (which is what libgpo uses). +Improved winbind logging and a new tool for parsing the winbind logs +-------------------------------------------------------------------- + +Winbind logs (if smb.conf 'winbind debug traceid = yes' is set) contain new +trace header fields 'traceid' and 'depth'. Field 'traceid' allows to track the +trace records belonging to the same request. Field 'depth' allows to track the +request nesting level. A new tool samba-log-parser is added for better log +parsing. + REMOVED FEATURES ================ @@ -45,6 +54,7 @@ smb.conf changes Parameter Name Description Default -------------- ----------- ------- + winbind debug traceid Add traceid No KNOWN ISSUES diff --git a/docs-xml/manpages/samba-log-parser.1.xml b/docs-xml/manpages/samba-log-parser.1.xml new file mode 100644 index 00000000000..ea6fd9150df --- /dev/null +++ b/docs-xml/manpages/samba-log-parser.1.xml @@ -0,0 +1,147 @@ +<?xml version="1.0" encoding="utf-8"?> +<!DOCTYPE refentry PUBLIC "-//Samba-Team//DTD DocBook V4.2-Based Variant V1.0//EN" "http://www.samba.org/samba/DTD/samba-doc"> +<refentry id="samba-log-parser.1"> + +<refmeta> + <refentrytitle>samba-log-parser</refentrytitle> + <manvolnum>1</manvolnum> + <refmiscinfo class="source">Samba</refmiscinfo> + <refmiscinfo class="manual">User Commands</refmiscinfo> + <refmiscinfo class="version">&doc.version;</refmiscinfo> +</refmeta> + + +<refnamediv> + <refname>samba-log-parser</refname> + <refpurpose>Samba (winbind) trace parser.</refpurpose> +</refnamediv> +options: + -h, --help show this help message and exit + --traceid ID specify the traceid of the trace records + --pid PID specify the pid of winbind client + --breakdown breakdown the traces into per traceid files + --merge merge logs by timestamp + --flow show the request/sub-request flow traces + --flow-compact show the request/sub-request flow traces without dcerpc details + +<refsynopsisdiv> + <cmdsynopsis> + <command>samba-log-parser</command> + <arg choice="req">path</arg> + <arg choice="opt">--pid=PID</arg> + <arg choice="opt">--traceid=ID</arg> + <arg choice="opt">--breakdown</arg> + <arg choice="opt">--merge</arg> + <arg choice="opt">--flow</arg> + <arg choice="opt">--flow-compact</arg> + <arg choice="opt">-h|--help</arg> + </cmdsynopsis> +</refsynopsisdiv> + +<refsect1> + <title>DESCRIPTION</title> + + <para>This tool is part of the <citerefentry><refentrytitle>samba</refentrytitle> + <manvolnum>7</manvolnum></citerefentry> suite.</para> + + <para>The <command>samba-log-parser</command> program parses samba winbind + logs.</para> +</refsect1> + + +<refsect1> + <title>OPTIONS</title> + + <para>The following options are available to the <command>samba-log-parser</command> program. + </para> + + <variablelist> + <varlistentry> + <term>--pid=PID</term> + <listitem><para>Display traces for winbind client with the matching PID. + </para></listitem> + </varlistentry> + + <varlistentry> + <term>--traceid=ID</term> + <listitem><para>Display traces with matching traceid debug header field. + </para></listitem> + </varlistentry> + + <varlistentry> + <term>--breakdown</term> + <listitem><para>Break down all traces to separate files in the current + working directory. For each traceid, three files are created: + traceid.full + traceid.flow + traceid.flowcompact + </para></listitem> + </varlistentry> + + <varlistentry> + <term>--merge</term> + <listitem><para>Sort the trace lines according to the timestamp. + </para></listitem> + </varlistentry> + + + <varlistentry> + <term>--flow</term> + <listitem><para>Display the request/sub-request flow. + </para></listitem> + </varlistentry> + + <varlistentry> + <term>--flow-compact</term> + <listitem><para>Display the request/sub-request flow without dcerpc + call details. + </para></listitem> + </varlistentry> + + </variablelist> +</refsect1> + + +<refsect1> + <title>EXAMPLES</title> + + <para>Show the flow traces for trace id + <parameter>1234</parameter> from log file log.winbind: + </para> + <programlisting> + # samba-log-parser --traceid 1234 --flow /var/log/samba/log.winbind + </programlisting> + + <para>Show the full traces for winbind client with PID + <parameter>999999</parameter> + sorted using the timestamp for log files found in the samba log directory: + </para> + + <programlisting> + # samba-log-parser --pid 999999 --merge /var/log/samba + </programlisting> + + <para>Break down the traces into separate files according to traceid sorted + using the timestamp for log files found in the samba log directory: + </para> + + <programlisting> + # samba-log-parser --breakdown --merge /var/log/samba + </programlisting> +</refsect1> + +<refsect1> + <title>VERSION</title> + + <para>This man page is part of version &doc.version; of the Samba suite.</para> +</refsect1> + +<refsect1> + <title>AUTHOR</title> + <para>The original Samba software and related utilities + were created by Andrew Tridgell. Samba is now developed + by the Samba Team as an Open Source project similar + to the way the Linux kernel is developed.</para> +</refsect1> + +</refentry> diff --git a/docs-xml/wscript_build b/docs-xml/wscript_build index aa4fc0a1254..95ed08ed1d8 100644 --- a/docs-xml/wscript_build +++ b/docs-xml/wscript_build @@ -53,6 +53,7 @@ manpages=''' manpages/traffic_replay.7 manpages/wbinfo.1 manpages/winbindd.8 + manpages/samba-log-parser.1 ''' pam_winbind_manpages = ''' diff --git a/source3/script/samba-log-parser b/source3/script/samba-log-parser new file mode 100755 index 00000000000..1c1d2ced33d --- /dev/null +++ b/source3/script/samba-log-parser @@ -0,0 +1,325 @@ +#!/usr/bin/env python3 +# +####################################################################### +# +# A script to parse samba (especially winbind) logfiles. +# Trace files should be in a non-syslog format (debug syslog format = no). +# +# --traceid ... Specify the traceid of the request to parse +# --pid ... Specify the pid +# --merge ... Merge logs by timestamp +# --flow ... Show the request/sub-request call flow +# --flow-compact ... Show the request/sub-request call flow without dcerpc +# +# +# Copyright (c) 2023 Andreas Schneider <a...@samba.org> +# Copyright (c) 2023 Pavel Filipenský <pfili...@redhat.com> +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +####################################################################### +# +# Requires: ??? + +import sys +import os +import re +from argparse import ArgumentParser +from collections import defaultdict + +# Trace record consists of a trace header followed by one or more text lines. +# Each trace header contains a traceid, which is the main identifier for this +# tool. A single traceid is either provided via command line option --traceid +# or a list of traceids is derived from the PID specified via option --pid. +# Creating and evaluating list of traceids from PID can be tricky: +# The traceid can appear in a trace record before trace record containing the +# PID is processed. So when we see a new traceid we are not sure if it belongs +# to the traced PID. +# The PID appears only in the main winbind process (log.winbind). If a +# directory with many log files should be processed, we process the files in +# random order. +# It might happen that e.g. log.wb-ADDOMAIN is processed before log.winbind so +# we do not know the list of traceids yet. +# To make all this easy we put into memory all trace records and do the final +# traceid filtering only after all files are read. This can require lot of +# memory if files are big. + + +def process_file(record_list, traceid_set, fname, opid, otraceid): + with open(fname, "r") as infile: + data = infile.readlines() + pid = None + traceid = 0 + traceid_prev = None + undecided_traceid = False + date = "" + record_lines = [] + + # If traceid option was provided the traceid_set will contain just it + if otraceid: + traceid_set.add(otraceid) + + RE_HEADER = re.compile( + r"^\[(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}\.\d{6}).*?, .*, " + r"traceid=([0-9]+).*\]") + RE_INTERFACE_VERSION = re.compile( + r"^\s+winbindd_interface_version: \[\S* \((\d+)\)\]") + RE_ASYNC_REQUEST = re.compile( + r"^\s+process_request_send: " + r"\[\S* \((\d+)\)\] Handling async request:") + # Example of a header line + # [2023/05/01 07:40:45.439049, 3, pid=418844, effective(0, 0), real(0, 0), class=winbind, traceid=37] ../../source3/winbindd/winbindd_misc.c:355(winbindd_interface_version) + for line in data: + header = RE_HEADER.search(line) + if header: + # Append all previous trace lines of a record if the traceid is in + # the list. + if record_lines: + record_list.append((date, traceid, record_lines, fname)) + record_lines = [] + # Remember the new date and the new traceid + date = header.group(1) + traceid = header.group(2) + if traceid != traceid_prev: + traceid_prev = traceid + undecided_traceid = True + if opid: + # Search for lines that identify a new winbind client and the + # client PID + + # winbindd_interface_version: [nss_winbind (500725)]: request interface version (version = 32) + # process_request_send: [nss_winbind (500725)] Handling async request: SETPWENT + interface_version = RE_INTERFACE_VERSION.search(line) + async_request = RE_ASYNC_REQUEST.search(line) + if interface_version: + pid = interface_version.group(1) + if undecided_traceid: + if pid == opid: + traceid_set.add(traceid) + undecided_traceid = False + if async_request: + pid = async_request.group(1) + if undecided_traceid: + if pid == opid: + traceid_set.add(traceid) + undecided_traceid = False + # For --breakdown add every traceid + if not opid and not otraceid: + traceid_set.add(traceid) + + record_lines.append(line) + + +def filter_traceids(record_list, traceid_set): + llist = [] + for (d, t, li, f) in record_list: + if t in traceid_set: + llist.append((d, t, li, f)) + return llist + + +def filter_flow(record_list): + local_list = [] + for (date, traceid, lines, filename) in record_list: + for line in lines: + isflow = re.search(r"^(\s+)flow: (.*)", line) + if isflow: + local_list.append(isflow.group(1) + isflow.group(2)) + return local_list + + +def filter_flowcompact(flist): + local_list = [] + end_marker = None + for fl in flist: + if not end_marker: + local_list.append(fl) + dcerpc_start = re.search(r"^(\s+)-> dcerpc_", fl) + if dcerpc_start: + end_marker = dcerpc_start.group(1) + else: + dcerpc_end = re.search(r"^" + end_marker + "<- dcerpc_", fl) + if dcerpc_end: + end_marker = None + local_list.append(fl) + return local_list + + +def print_record_list(record_list, file): + f_prev = None + for (date, traceid, lines, filename) in record_list: + # Inform about filename change + if filename != f_prev: + print("-" * 72, file=file) + print("FILE: ", filename, file=file) + print("-" * 72, file=file) + for line in lines: + print(line, end='', file=file) + f_prev = filename + +# record_list ... list of quadruplets <date, traceid, [trace lines], filename> +# flow_list ... lines from record_list with 'flow' traces +# traceid_set ... list of traceids we want to trace +# with --traceid ... there is a single traceids +# with --pid ... there are all traceids for the PID +# with --breakdown ... there are all traceids + + +def setup_parser(): + parser = ArgumentParser() + + parser.add_argument( + "path", + type=str, + help="logfile or directory" + ) + parser.add_argument( + "--traceid", + dest="traceid", + help="specify the traceid of the trace records", + metavar="ID" + ) + parser.add_argument( + "--pid", + dest="pid", + help="specify the pid of winbind client", + metavar="PID" + ) + parser.add_argument( + "--breakdown", + action="store_true", + dest="breakdown", + default=False, + help="breakdown the traces into per traceid files" + ) + parser.add_argument( + "--merge", + action="store_true", + dest="merge", + default=False, + help="merge logs by timestamp" + ) + parser.add_argument( + "--flow", + action="store_true", + dest="flow", + default=False, + help="show the request/sub-request flow traces" + ) + parser.add_argument( + "--flow-compact", + action="store_true", + dest="flowcompact", + default=False, + help="show the request/sub-request flow traces without dcerpc details" + ) + return parser + + +def main(): # noqa + record_list = [] + flow_list = [] + traceid_set = set() + + parser = setup_parser() + options = parser.parse_args() + + if not options.traceid and not options.pid and not options.breakdown: + print("One of --traceid or --pid is needed or --breakdown.") + sys.exit(1) + elif options.traceid and options.pid: + print("Only one of --traceid or --pid or --breakdown is allowed.") + sys.exit(1) + + if options.flow and not options.traceid: + print("Option --flow can be used only together with --traceid.") + sys.exit(1) + + if options.flowcompact and not options.traceid: + print("Option --flow-compact can be used only together with " + "--traceid.") + sys.exit(1) + + if options.flow and options.flowcompact: + print("Only one of --flow or --flow-compact is allowed.") + sys.exit(1) + + if not options.path: + print("Path to logfile or directory with logs is needed.") + sys.exit(1) + + path = options.path + if os.path.isdir(path): + for root, dirs, files in os.walk(path): + for name in files: + process_file( + record_list, + traceid_set, + os.path.join(root, name), + options.pid, + options.traceid, + ) + elif os.path.isfile(path): + process_file( + record_list, + traceid_set, + path, + options.pid, + options.traceid + ) + else: + print(path, "Path is neither file or directory.") + sys.exit(1) + + # Keep only records with matching traceids + if not options.breakdown: + record_list = filter_traceids(record_list, traceid_set) + + if options.breakdown: + for traceid in traceid_set: + # Full + with open("%s.full" % traceid, "w") as full_f: + full_l = filter_traceids(record_list, {traceid}) + if options.merge: + full_l.sort() + print_record_list(full_l, full_f) + # Flow + with open("%s.flow" % traceid, "w") as flow_f: + flow_l = filter_flow(full_l) + for fl in flow_l: + print(fl, file=flow_f) -- Samba Shared Repository