Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package perl-Cavil-Matcher for openSUSE:Factory checked in at 2026-09-08 16:58:14 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/perl-Cavil-Matcher (Old) and /work/SRC/openSUSE:Factory/.perl-Cavil-Matcher.new.1265 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "perl-Cavil-Matcher" Tue Sep 8 16:58:14 2026 rev:3 rq:1376326 version:1.50.0 Changes: -------- --- /work/SRC/openSUSE:Factory/perl-Cavil-Matcher/perl-Cavil-Matcher.changes 2026-08-06 16:28:59.290628970 +0200 +++ /work/SRC/openSUSE:Factory/.perl-Cavil-Matcher.new.1265/perl-Cavil-Matcher.changes 2026-09-08 17:01:54.726486500 +0200 @@ -1,0 +2,45 @@ +Fri Sep 4 09:40:56 UTC 2026 - Tina Müller <[email protected]> + +- updated to 1.50.0 (1.05) + see /usr/share/doc/packages/perl-Cavil-Matcher/Changes + + 1.05 2026-09-03 + - Removed the on-disk fingerprint index (fp_build/fp_open, the Cavil::Matcher::Fingerprints + package, and Cavil::Matcher::FpIndex); consumers now keep the searchable index in their own + database. The fingerprint_file and content_hash primitives are unchanged. + 1.04 2026-08-27 + - FpIndex now caches opened segments and reuses them across searches instead of reopening every + segment on every query. Opening a segment is a fixed per-query cost that otherwise dominated search + time on a long-lived query server, regardless of the query. Segments are immutable append-only + files, so a cached handle always maps the same bytes; a handle whose segment has been compacted out + of the manifest is dropped on the next search. Results are unchanged. + - score() and FpIndex::search() take an optional minimum containment; matches below it are dropped + inside the scorer instead of being returned for the caller to filter. A query dominated by common + fingerprints can match hundreds of thousands of contents, almost all coincidental, so applying the + floor at this level avoids building and returning them. Defaults to 0 (return everything). + - score() and FpIndex::search() also take an optional max_df: query fingerprints appearing in more + than that many records of a segment are ignored (boilerplate that matches nearly everything, adding + only coincidental hits and skewing containment). Off by default (0). Intended for a compacted + single-segment index, where a segment's record count for a fingerprint is its document frequency. + 1.03 2026-08-27 + - Fingerprint scoring now reports, for each matched fingerprint, the query fingerprint value it was, + appended to each region tuple after the existing start line and span. This lets a caller map + matches back to their query positions - enough to tell an aligned copy from scattered coincidental + hits, and to derive each query fingerprint's document frequency from a full result set. Backward + compatible: the (start_line, span) pair at the front of every region tuple is unchanged. + 1.02 2026-08-27 + - Snippet code search: a new winnowed-fingerprint index for "where does this code already exist" + provenance queries, alongside the license-pattern matcher. fingerprint_file winnows a file's + tokens (k tokens per gram, w grams per window) into content fingerprints; a fingerprint segment + stores them keyed by a 128-bit content hash rather than filenames (so it stays lean and joins to + the Cavil database), and scoring ranks candidates by both-direction containment with exact + matched-line spans for highlighting. + - Cavil::Matcher::FpIndex is the pure-Perl lifecycle over a directory of fingerprint segments + (incremental add_segment, manifest/generation, memory-mapped shared reads), reusing the same + on-disk discipline as the pattern index: versioned, CRC-checksummed, structure-validated on + every open, with corrupt or foreign files rejected rather than mis-read. + - Byte-identical files are de-duplicated by content hash, so the same source shipped across many + package versions is stored once, with no package-name parsing. + - Additive only: the license-pattern format is unchanged and no migration is needed. + +------------------------------------------------------------------- Old: ---- Cavil-Matcher-1.01.tar.gz New: ---- Cavil-Matcher-1.05.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ perl-Cavil-Matcher.spec ++++++ --- /var/tmp/diff_new_pack.X8xdmx/_old 2026-09-08 17:01:55.467517475 +0200 +++ /var/tmp/diff_new_pack.X8xdmx/_new 2026-09-08 17:01:55.469517558 +0200 @@ -18,10 +18,10 @@ %define cpan_name Cavil-Matcher Name: perl-Cavil-Matcher -Version: 1.10.0 +Version: 1.50.0 Release: 0 -# 1.01 -> normalize -> 1.10.0 -%define cpan_version 1.01 +# 1.05 -> normalize -> 1.50.0 +%define cpan_version 1.05 License: GPL-1.0-or-later Summary: Next-generation license pattern matcher for Cavil URL: https://metacpan.org/release/%{cpan_name} ++++++ Cavil-Matcher-1.01.tar.gz -> Cavil-Matcher-1.05.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/Changes new/Cavil-Matcher-1.05/Changes --- old/Cavil-Matcher-1.01/Changes 2026-08-04 20:28:44.956802618 +0200 +++ new/Cavil-Matcher-1.05/Changes 2026-09-03 12:55:39.857161295 +0200 @@ -1,5 +1,47 @@ Revision history for Cavil::Matcher +1.05 2026-09-03 + - Removed the on-disk fingerprint index (fp_build/fp_open, the Cavil::Matcher::Fingerprints + package, and Cavil::Matcher::FpIndex); consumers now keep the searchable index in their own + database. The fingerprint_file and content_hash primitives are unchanged. + +1.04 2026-08-27 + - FpIndex now caches opened segments and reuses them across searches instead of reopening every + segment on every query. Opening a segment is a fixed per-query cost that otherwise dominated search + time on a long-lived query server, regardless of the query. Segments are immutable append-only + files, so a cached handle always maps the same bytes; a handle whose segment has been compacted out + of the manifest is dropped on the next search. Results are unchanged. + - score() and FpIndex::search() take an optional minimum containment; matches below it are dropped + inside the scorer instead of being returned for the caller to filter. A query dominated by common + fingerprints can match hundreds of thousands of contents, almost all coincidental, so applying the + floor at this level avoids building and returning them. Defaults to 0 (return everything). + - score() and FpIndex::search() also take an optional max_df: query fingerprints appearing in more + than that many records of a segment are ignored (boilerplate that matches nearly everything, adding + only coincidental hits and skewing containment). Off by default (0). Intended for a compacted + single-segment index, where a segment's record count for a fingerprint is its document frequency. + +1.03 2026-08-27 + - Fingerprint scoring now reports, for each matched fingerprint, the query fingerprint value it was, + appended to each region tuple after the existing start line and span. This lets a caller map + matches back to their query positions - enough to tell an aligned copy from scattered coincidental + hits, and to derive each query fingerprint's document frequency from a full result set. Backward + compatible: the (start_line, span) pair at the front of every region tuple is unchanged. + +1.02 2026-08-27 + - Snippet code search: a new winnowed-fingerprint index for "where does this code already exist" + provenance queries, alongside the license-pattern matcher. fingerprint_file winnows a file's + tokens (k tokens per gram, w grams per window) into content fingerprints; a fingerprint segment + stores them keyed by a 128-bit content hash rather than filenames (so it stays lean and joins to + the Cavil database), and scoring ranks candidates by both-direction containment with exact + matched-line spans for highlighting. + - Cavil::Matcher::FpIndex is the pure-Perl lifecycle over a directory of fingerprint segments + (incremental add_segment, manifest/generation, memory-mapped shared reads), reusing the same + on-disk discipline as the pattern index: versioned, CRC-checksummed, structure-validated on + every open, with corrupt or foreign files rejected rather than mis-read. + - Byte-identical files are de-duplicated by content hash, so the same source shipped across many + package versions is stored once, with no package-name parsing. + - Additive only: the license-pattern format is unchanged and no migration is needed. + 1.01 2026-08-04 - Big indexing speedup: the scan path (load/attach) no longer re-checksums a segment's whole payload on every open. A compiled segment is an immutable, atomically-published cache that is diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/MANIFEST new/Cavil-Matcher-1.05/MANIFEST --- old/Cavil-Matcher-1.01/MANIFEST 2026-08-04 20:33:35.156164233 +0200 +++ new/Cavil-Matcher-1.05/MANIFEST 2026-09-03 12:57:59.858246039 +0200 @@ -13,6 +13,8 @@ src/bag.h src/bindings.cc src/bindings.h +src/fingerprint.cc +src/fingerprint.h src/matcher.cc src/matcher.h src/segment.cc @@ -36,6 +38,7 @@ t/14coverage.t t/15segment_header.t t/16resolve.t +t/fingerprint.t t/fixtures/licenses/04license.1.pattern t/fixtures/licenses/04license.1.txt t/fixtures/licenses/04license.10.pattern diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/MANIFEST.SKIP new/Cavil-Matcher-1.05/MANIFEST.SKIP --- old/Cavil-Matcher-1.01/MANIFEST.SKIP 2026-07-22 14:59:22.000000000 +0200 +++ new/Cavil-Matcher-1.05/MANIFEST.SKIP 2026-08-27 13:20:49.000000000 +0200 @@ -1,5 +1,7 @@ ^blib/ ^Matcher\.c$ +^tools/ +^MANIFEST\.bak$ ^Makefile$ ^Makefile\.old$ ^pm_to_blib diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/META.json new/Cavil-Matcher-1.05/META.json --- old/Cavil-Matcher-1.01/META.json 2026-08-04 20:33:35.142118638 +0200 +++ new/Cavil-Matcher-1.05/META.json 2026-09-03 12:57:59.848143644 +0200 @@ -60,6 +60,6 @@ "web" : "https://github.com/openSUSE/cavil-matcher"; } }, - "version" : "1.01", + "version" : "1.05", "x_serialization_backend" : "JSON::PP version 4.16" } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/META.yml new/Cavil-Matcher-1.05/META.yml --- old/Cavil-Matcher-1.01/META.yml 2026-08-04 20:33:35.110368413 +0200 +++ new/Cavil-Matcher-1.05/META.yml 2026-09-03 12:57:59.814717384 +0200 @@ -28,5 +28,5 @@ bugtracker: https://github.com/openSUSE/cavil-matcher/issues license: https://www.gnu.org/licenses/old-licenses/gpl-2.0.html repository: https://github.com/openSUSE/cavil-matcher.git -version: '1.01' +version: '1.05' x_serialization_backend: 'CPAN::Meta::YAML version 0.020' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/Makefile.PL new/Cavil-Matcher-1.05/Makefile.PL --- old/Cavil-Matcher-1.01/Makefile.PL 2026-07-21 16:19:43.063785774 +0200 +++ new/Cavil-Matcher-1.05/Makefile.PL 2026-09-03 12:43:00.243958370 +0200 @@ -14,7 +14,7 @@ if $Config{uvsize} < 8; # C++ core lives under src/; the XS shim (Matcher.xs) at the top marshals between Perl and it. -my @cpp = qw(SpookyV2.cpp tokenizer.cc segment.cc matcher.cc bag.cc bindings.cc); +my @cpp = qw(SpookyV2.cpp tokenizer.cc segment.cc matcher.cc bag.cc fingerprint.cc bindings.cc); my @obj = ('$(BASEEXT)$(OBJ_EXT)'); # the compiled XS shim (Matcher$(OBJ_EXT)) for my $src (@cpp) { (my $o = $src) =~ s/\.(?:cc|cpp)$/\$(OBJ_EXT)/; @@ -57,8 +57,9 @@ clean => {FILES => 'cover_db'}, depend => { - 'src/bindings$(OBJ_EXT)' => 'src/bindings.h src/tokenizer.h src/segment.h src/matcher.h src/bag.h src/SpookyV2.h', + 'src/bindings$(OBJ_EXT)' => 'src/bindings.h src/tokenizer.h src/segment.h src/matcher.h src/bag.h src/fingerprint.h src/SpookyV2.h', 'src/matcher$(OBJ_EXT)' => 'src/matcher.h src/segment.h src/tokenizer.h', + 'src/fingerprint$(OBJ_EXT)' => 'src/fingerprint.h src/tokenizer.h src/SpookyV2.h', 'src/segment$(OBJ_EXT)' => 'src/segment.h src/tokenizer.h', 'src/tokenizer$(OBJ_EXT)' => 'src/tokenizer.h src/SpookyV2.h', 'src/bag$(OBJ_EXT)' => 'src/bag.h src/tokenizer.h', diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/Matcher.xs new/Cavil-Matcher-1.05/Matcher.xs --- old/Cavil-Matcher-1.01/Matcher.xs 2026-08-04 20:27:15.064991701 +0200 +++ new/Cavil-Matcher-1.05/Matcher.xs 2026-09-03 12:34:18.652668444 +0200 @@ -56,6 +56,18 @@ OUTPUT: RETVAL +AV *fingerprint_file(const char *path, int k, int w) + CODE: + RETVAL = fp_fingerprint_file(path, k, w); + OUTPUT: + RETVAL + +SV *content_hash(const char *path) + CODE: + RETVAL = fp_content_hash(path); + OUTPUT: + RETVAL + MODULE = Cavil::Matcher PACKAGE = Cavil::Matcher::Engine void add_pattern(Cavil::Matcher::Engine self, UV id, AV *tokens) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/docs/Architecture.md new/Cavil-Matcher-1.05/docs/Architecture.md --- old/Cavil-Matcher-1.01/docs/Architecture.md 2026-08-04 20:20:04.757907627 +0200 +++ new/Cavil-Matcher-1.05/docs/Architecture.md 2026-09-03 12:45:21.340350699 +0200 @@ -7,7 +7,7 @@ ## Why this exists Cavil reviews the licensing of software by scanning source code for the text of known licenses. The scanning -is done by matching every file against a large, ever-growing collection of *patterns* — normalized fragments +is done by matching every file against a large, ever-growing collection of *patterns* - normalized fragments of license text. The previous engine did this well and fast, and this one keeps its core idea unchanged. What it changes is everything around that core, because the old design had three operational problems that grow worse over time: @@ -20,8 +20,8 @@ - **The on-disk form was fragile.** The compiled file had no header, version, or checksum; it was trusted blindly. A format change or a truncated file could be silently misread. -The goal of this engine is to fix those three things — cheap incremental updates, one shared copy per machine, -and a safe, versioned on-disk format — while keeping matching itself bit-for-bit identical, so switching to it +The goal of this engine is to fix those three things - cheap incremental updates, one shared copy per machine, +and a safe, versioned on-disk format - while keeping matching itself bit-for-bit identical, so switching to it requires no re-processing of existing data. ## The Perl/native split, and why @@ -30,8 +30,8 @@ the pattern collection. That inner loop, and the hashing that feeds it, is where nearly all the time goes, and it is written in a small, carefully-frozen C++ core. -Everything else — deciding which patterns are active, recording that a pattern was removed, choosing when to -compact, reading and writing the little file that describes the collection — happens rarely and on small data. +Everything else - deciding which patterns are active, recording that a pattern was removed, choosing when to +compact, reading and writing the little file that describes the collection - happens rarely and on small data. All of that is plain Perl, because that is where the team is strongest, because it is where a newcomer can follow the logic without a debugger, and because none of it is on the hot path, so nothing is lost by keeping it in Perl. The native side is deliberately dumb: it is handed a list of things to search and simply searches @@ -40,7 +40,7 @@ The native core is C++ rather than a rewrite in another language for a simple reason: the matching algorithm and its hashing are already proven on a legal tool, where a subtle behavioural change is the worst kind of bug. Reusing that code unchanged is the safest possible choice, so the core stays in the language it is -already written in. The genuinely new native code — the reader for the on-disk format — is small and only ever +already written in. The genuinely new native code - the reader for the on-disk format - is small and only ever reads files this same software wrote, and it validates everything it reads, which removes the one real weakness the old format had. @@ -50,7 +50,7 @@ and the files being scanned are put through the same normalization: text is lower-cased and split into words, punctuation and common comment or markup noise is discarded, and each surviving word is reduced to a number. Only those numbers are ever compared. This is what lets a match survive reformatting, rewrapping, and -different comment styles — the layout simply disappears during normalization. +different comment styles - the layout simply disappears during normalization. One wildcard exists. A pattern may say "skip one to N words here" (at least one word, at most N - it does not match a zero-word gap), which lets a single pattern absorb the parts of a license that legitimately vary, such @@ -85,7 +85,7 @@ not touched. Absorbing a new pattern is therefore cheap and local, no matter how large the collection has grown. - **Removing a pattern writes a tombstone** in the manifest and nothing else. At scan time, matches belonging - to a tombstoned pattern are dropped before overlap resolution — so removing a pattern correctly reveals any + to a tombstoned pattern are dropped before overlap resolution - so removing a pattern correctly reveals any smaller matches it had been hiding, exactly as if it had never existed. No segment is recompiled. - **A query searches all active segments at once**, gathers their matches, discards the tombstoned ones, and then applies the ordinary overlap resolution to the combined set. The result is identical to what a single @@ -93,7 +93,7 @@ Because deltas and tombstones accumulate, an occasional **compaction** folds the current pattern set back into a single fresh base segment and clears the tombstones. This is the one operation that reads the full pattern -set from the database — Cavil's source of truth — and it is rare and runs in the background, off the scanning +set from the database - Cavil's source of truth - and it is rare and runs in the background, off the scanning path. It exists only to keep the number of segments and the length of the tombstone list bounded over time. ## Shared memory and reproducibility @@ -106,14 +106,14 @@ The generation number in the manifest gives reproducibility. A scan pins the generation it ran against, and a report can record it, so re-running an old report can use exactly the same patterns it originally saw. Updates -are published atomically — a new segment and an updated manifest are written to the side and swapped into place -in one step — so a reader never observes a half-written collection. +are published atomically - a new segment and an updated manifest are written to the side and swapped into place +in one step - so a reader never observes a half-written collection. ## The on-disk format Each segment file begins with a header identifying it, stating its format version, and carrying a checksum of -everything that follows. Opening a segment always validates its **structure** — the header fields, the size, -and every internal reference — so a file that is truncated, of the wrong version, structurally impossible, or +everything that follows. Opening a segment always validates its **structure** - the header fields, the size, +and every internal reference - so a file that is truncated, of the wrong version, structurally impossible, or simply not a segment at all is rejected cleanly; it is never partially trusted and never able to send the scanner off the end of the data. This structural validation is cheap and always on, so memory safety never depends on the checksum. @@ -122,7 +122,7 @@ recomputed on the hot scan path. A segment is Cavil's own derived cache: it is checksummed when compiled, published atomically (written to the side and renamed into place), and thereafter immutable and regenerable from the database. Re-checksumming a multi-hundred-megabyte payload on every one of the thousands of indexing -opens — where the mmap already shares one physical copy — is almost the entire cost of a load (measured at +opens - where the mmap already shares one physical copy - is almost the entire cost of a load (measured at ~97%), and it guards against a corruption that atomic publishing already prevents. So the CRC is verified where it is meaningful: once when the segment is written, and on demand through the engine's `verify` entry point (an fsck for operators). The manifest additionally records a checksum for each segment for that on-demand check. @@ -135,7 +135,36 @@ single-line files with no structure at all. The matcher treats all of this as ordinary input: it reads files in bounded chunks, stops cleanly at the end of usable data, and bounds the amount of a file it holds in memory at once. Unreadable paths and missing files produce empty results rather than errors. The guiding rule is -simple and absolute — no input, however hostile or malformed, may crash the scan. +simple and absolute - no input, however hostile or malformed, may crash the scan. + +## Fingerprinting for snippet provenance + +The pattern engine answers "which known licenses does this file contain". A closely related question is +"which known open source code does this snippet resemble", used by a service where someone submits a +fragment (for example AI-generated code) and asks how much of it already exists in the open source Cavil +has seen. This distribution provides the *primitives* for that question. The searchable index that turns +fingerprints back into packages and paths lives in the consuming application - Cavil keeps it in Postgres - +not here. + +Two calls make up the surface. `content_hash` returns the 128-bit hash of a file's raw bytes as 32 hex +characters, produced by the same frozen hasher the pattern side uses, so identical content always yields +the same key and the database can join on it. `fingerprint_file` winnows a file into a set of fingerprints, +each carrying the exact line range it covers so a match can be highlighted. + +Winnowing is the idea plagiarism detectors have used for decades. A file's tokens are grouped into +overlapping runs (k tokens each), every run is hashed, and a stable subset of those hashes is selected by +taking the smallest hash in each sliding window of the runs. The selection is deterministic and +shift-stable, so the same code always yields the same fingerprints no matter where it sits in a file or how +it was chopped up while reading, and it survives reformatting and renaming because the layout has already +disappeared during tokenization. The window width is the one knob: a wider window keeps fewer fingerprints +(cheaper) and a narrower window keeps more (more robust), with the count landing near two divided by the +window-plus-one, times the number of runs. + +Everything downstream is the consumer's: storing the fingerprints, looking a query's fingerprints up, +ranking candidates by containment, pruning boilerplate, and resolving a content hash back to packages and +paths. Cavil does all of that in the database. Keeping only the primitives here means the expensive, +corpus-sized index is owned where it is queried, and this distribution stays a deterministic fingerprint +source with nothing on disk of its own to maintain. ## What deliberately stays the same @@ -175,5 +204,5 @@ can simply be deleted, and the self-contained suite stands on its own. A change to the pattern set flows through the system in the obvious way: adding patterns writes a new segment, -removing one writes a tombstone, and compaction periodically rewrites a single clean base — none of which +removing one writes a tombstone, and compaction periodically rewrites a single clean base - none of which disturbs the data an in-flight scan is already using. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/lib/Cavil/Matcher.pm new/Cavil-Matcher-1.05/lib/Cavil/Matcher.pm --- old/Cavil-Matcher-1.01/lib/Cavil/Matcher.pm 2026-08-04 20:28:57.000000000 +0200 +++ new/Cavil-Matcher-1.05/lib/Cavil/Matcher.pm 2026-09-03 12:45:54.000000000 +0200 @@ -6,7 +6,7 @@ use strict; use warnings; -our $VERSION = '1.01'; +our $VERSION = '1.05'; require XSLoader; XSLoader::load('Cavil::Matcher', $VERSION); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/src/bindings.cc new/Cavil-Matcher-1.05/src/bindings.cc --- old/Cavil-Matcher-1.01/src/bindings.cc 2026-08-04 20:27:02.193018589 +0200 +++ new/Cavil-Matcher-1.05/src/bindings.cc 2026-09-03 12:33:47.613296616 +0200 @@ -4,6 +4,7 @@ // Pure-C++ core first, so its <vector>/<map>/<string> are seen before Perl's macro soup. #include "SpookyV2.h" #include "bag.h" +#include "fingerprint.h" #include "matcher.h" #include "segment.h" #include "tokenizer.h" @@ -352,3 +353,28 @@ int bag_dump(Bag* b, const char* filename) { return b->dump(filename) ? 1 : 0; } int bag_load(Bag* b, const char* filename) { return b->load(filename, /*verify_crc=*/false) ? 1 : 0; } int bag_verify(Bag* b, const char* filename) { return b->verify(filename) ? 1 : 0; } + +// --------------------------------------------------------------------------- +// Fingerprints (snippet provenance) +// --------------------------------------------------------------------------- +AV* fp_fingerprint_file(const char* path, int k, int w) { + dTHX; + AV* ret = newAV(); + for (const Fingerprint& f : fingerprint_file(path, k, w)) { + AV* row = newAV(); + av_push(row, newSVuv(f.fp)); + av_push(row, newSVuv(f.sline)); + av_push(row, newSVuv(f.eline)); + av_push(ret, newRV_noinc((SV*)row)); + } + return ret; +} + +SV* fp_content_hash(const char* path) { + dTHX; + ContentHash h; + fingerprint_file(path, 1, 1, nullptr, &h); // k/w irrelevant; we only want the content hash + char buf[33]; + snprintf(buf, sizeof(buf), "%016llx%016llx", (unsigned long long)h.hi, (unsigned long long)h.lo); + return newSVpv(buf, 32); +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/src/bindings.h new/Cavil-Matcher-1.05/src/bindings.h --- old/Cavil-Matcher-1.01/src/bindings.h 2026-08-04 20:27:06.717009139 +0200 +++ new/Cavil-Matcher-1.05/src/bindings.h 2026-09-03 12:28:57.555999859 +0200 @@ -40,6 +40,10 @@ void matcher_set_generation(Matcher* m, UV generation); UV matcher_generation(Matcher* m); +// Fingerprints (snippet provenance): winnow a file, or hash its content. +AV* fp_fingerprint_file(const char* path, int k, int w); // [[fp, sline, eline], ...] +SV* fp_content_hash(const char* path); // 32-hex content hash + // Bag Bag* pattern_init_bag(); void destroy_bag(Bag* b); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/src/fingerprint.cc new/Cavil-Matcher-1.05/src/fingerprint.cc --- old/Cavil-Matcher-1.01/src/fingerprint.cc 1970-01-01 01:00:00.000000000 +0100 +++ new/Cavil-Matcher-1.05/src/fingerprint.cc 2026-09-03 12:24:38.111343093 +0200 @@ -0,0 +1,97 @@ +// SPDX-FileCopyrightText: SUSE LLC +// SPDX-License-Identifier: GPL-2.0-or-later + +#include "fingerprint.h" + +#include "SpookyV2.h" // content hashing + +#include <cstdio> +#include <cstring> +#include <vector> + +static const int MAX_LINE_SIZE = 8000; + +// A single file that winnows to more than this many tokens is almost certainly a minified/generated +// blob, not human source; cap it so one pathological file cannot balloon memory. +static const size_t MAX_TOKENS_PER_FILE = 4000000; + +std::vector<Fingerprint> winnow_tokens(const TokenList& tokens, int k, int w) { + std::vector<Fingerprint> out; + if (k < 1) k = 1; + if (w < 1) w = 1; + int n = (int)tokens.size(); + int ngrams = n - k + 1; + if (ngrams <= 0) return out; + + // Hash each k-gram of token hashes (files never contain $SKIP tokens, so every hash is real). + std::vector<uint64_t> g((size_t)ngrams); + for (int i = 0; i < ngrams; ++i) { + uint64_t h = 1469598103934665603ULL; + for (int j = 0; j < k; ++j) { + h ^= tokens[(size_t)i + j].hash; + h *= 1099511628211ULL; + } + g[(size_t)i] = h; + } + + // Classic winnowing (Schleimer/Wilkerson/Aiken): emit the minimum hash of every window of w grams, + // preferring the rightmost on ties, never re-emitting the same selected position. A gram at position p + // spans tokens p..p+k-1, so its line range is those tokens' line numbers. + auto emit = [&](int pos) { + out.push_back({g[(size_t)pos], (uint32_t)tokens[(size_t)pos].linenumber, + (uint32_t)tokens[(size_t)(pos + k - 1)].linenumber}); + }; + int min_pos = -1; + for (int i = 0; i + w <= ngrams; ++i) { + if (min_pos < i) { + min_pos = i; + for (int j = i + 1; j < i + w; ++j) + if (g[(size_t)j] <= g[(size_t)min_pos]) min_pos = j; + emit(min_pos); + } else if (g[(size_t)(i + w - 1)] <= g[(size_t)min_pos]) { + min_pos = i + w - 1; + emit(min_pos); + } + } + return out; +} + +// The shared read+tokenize+winnow path, byte-for-byte the same numbering as Matcher::find_matches, so a +// fingerprint's line spans line up with a pattern match's. When `content` is non-null the raw bytes are +// hashed as read, before tokenize lower-cases them, yielding the content hash the caller joins on. +static std::vector<Fingerprint> winnow_stream(FILE* input, int k, int w, size_t* out_tokens, + SpookyHash* content) { + TokenList ts; + char line[MAX_LINE_SIZE]; + int linenumber = 1; + long pos = ftell(input); + while (fgets(line, sizeof(line) - 1, input)) { + long npos = ftell(input); + size_t got = (pos >= 0 && npos >= pos) ? (size_t)(npos - pos) : strlen(line); + pos = npos; + if (got >= sizeof(line)) got = strlen(line); + if (content) content->Update(line, got); + bool line_end = got > 0 && line[got - 1] == '\n'; + tokenizer().tokenize(ts, line, linenumber); + if (line_end) ++linenumber; + if (ts.size() > MAX_TOKENS_PER_FILE) break; + } + if (out_tokens) *out_tokens = ts.size(); + return winnow_tokens(ts, k, w); +} + +std::vector<Fingerprint> fingerprint_file(const std::string& path, int k, int w, size_t* out_tokens, + ContentHash* out_hash) { + if (out_tokens) *out_tokens = 0; + if (out_hash) *out_hash = ContentHash{}; + FILE* input = fopen(path.c_str(), "rb"); + if (!input) return {}; + + SpookyHash content; + if (out_hash) content.Init(0, 0); + auto fps = winnow_stream(input, k, w, out_tokens, out_hash ? &content : nullptr); + fclose(input); + + if (out_hash) content.Final(&out_hash->hi, &out_hash->lo); + return fps; +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/src/fingerprint.h new/Cavil-Matcher-1.05/src/fingerprint.h --- old/Cavil-Matcher-1.01/src/fingerprint.h 1970-01-01 01:00:00.000000000 +0100 +++ new/Cavil-Matcher-1.05/src/fingerprint.h 2026-09-03 12:28:46.703597230 +0200 @@ -0,0 +1,47 @@ +// SPDX-FileCopyrightText: SUSE LLC +// SPDX-License-Identifier: GPL-2.0-or-later +// +// Snippet-provenance primitives: winnow a file's tokens into content fingerprints, and hash its bytes +// into a 128-bit content key. These answer "what fingerprints does this content have"; the searchable +// index that turns fingerprints back into packages/paths lives in the consuming application (Cavil, in +// Postgres), not here. Uses the frozen tokenizer/hashing so identical text always fingerprints +// identically. No Perl types here. + +#ifndef CAVIL_MATCHER_FINGERPRINT_H_ +#define CAVIL_MATCHER_FINGERPRINT_H_ + +#include "tokenizer.h" + +#include <cstdint> +#include <string> +#include <vector> + +// A 128-bit content hash (SpookyHash of the raw file bytes), rendered elsewhere as 32 hex chars exactly +// like Cavil::Matcher::Hash::hex, so the Cavil database can join on it. +struct ContentHash { + uint64_t hi = 0; + uint64_t lo = 0; + bool operator==(const ContentHash& o) const { return hi == o.hi && lo == o.lo; } +}; + +// One winnowed fingerprint occurrence: the 64-bit gram hash and the source line range (start..end) it +// covers, so a match can be highlighted exactly. +struct Fingerprint { + uint64_t fp; + uint32_t sline; + uint32_t eline; +}; + +// Winnow a token stream into fingerprints. k = tokens per gram, w = grams per window; expected density +// is ~2/(w+1) of grams. Deterministic (rightmost-min tie-break), so identical text always yields +// identical fingerprints regardless of how the file was chunked while reading. +std::vector<Fingerprint> winnow_tokens(const TokenList& tokens, int k, int w); + +// Fingerprint a whole file: read it (raw bytes, NUL-tolerant, token count bounded like the matcher), +// tokenize, winnow. out_tokens receives the token count; out_hash receives the 128-bit content hash the +// caller joins on (identical across byte-identical files). Missing or unreadable files yield an empty +// result, not an error. +std::vector<Fingerprint> fingerprint_file(const std::string& path, int k, int w, size_t* out_tokens = nullptr, + ContentHash* out_hash = nullptr); + +#endif diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/t/fingerprint.t new/Cavil-Matcher-1.05/t/fingerprint.t --- old/Cavil-Matcher-1.01/t/fingerprint.t 1970-01-01 01:00:00.000000000 +0100 +++ new/Cavil-Matcher-1.05/t/fingerprint.t 2026-09-03 12:37:29.000000000 +0200 @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: SUSE LLC +# SPDX-License-Identifier: GPL-2.0-or-later + +use strict; +use warnings; +use Test::More; +use Cavil::Matcher; +use File::Temp qw(tempdir); +use File::Spec; + +# Small k/w so tiny fixtures still winnow to fingerprints (density ~2/(w+1)). +my ($K, $W) = (3, 4); +my $dir = tempdir(CLEANUP => 1); + +sub write_file { + my ($name, $text) = @_; + my $path = File::Spec->catfile($dir, $name); + open my $fh, '>', $path or die "write $path: $!"; + print $fh $text; + close $fh; + return $path; +} + +my $block_a = join ' ', map {"alpha$_ beta$_ gamma$_ delta$_"} 1 .. 20; +my $block_b = join ' ', map {"omega$_ sigma$_ kappa$_ lambda$_"} 1 .. 20; + +my $file_a = write_file('a.txt', $block_a); +my $file_b = write_file('b.txt', $block_b); + +my ($hash_a, $hash_b) = map { Cavil::Matcher::content_hash($_) } $file_a, $file_b; + +subtest 'content_hash is a stable 32-hex digest of the bytes' => sub { + like $hash_a, qr/^[0-9a-f]{32}$/, '32 hex chars'; + is $hash_a, Cavil::Matcher::content_hash($file_a), 'same file, same hash'; + isnt $hash_a, $hash_b, 'different content, different hash'; +}; + +subtest 'fingerprint_file is deterministic with exact line spans' => sub { + my $fp1 = Cavil::Matcher::fingerprint_file($file_a, $K, $W); + my $fp2 = Cavil::Matcher::fingerprint_file($file_a, $K, $W); + ok @$fp1 > 0, 'produced fingerprints'; + is_deeply $fp2, $fp1, 'identical input yields identical fingerprints'; + is scalar(grep { @$_ == 3 && $_->[2] >= $_->[1] } @$fp1), scalar(@$fp1), 'each row is [fp, sline, eline]'; +}; + +subtest 'fingerprint_file on missing/empty input never dies' => sub { + is_deeply Cavil::Matcher::fingerprint_file(File::Spec->catfile($dir, 'nope.txt'), $K, $W), [], + 'missing file => empty'; + is_deeply Cavil::Matcher::fingerprint_file(write_file('empty.txt', ''), $K, $W), [], 'empty file => empty'; + is_deeply Cavil::Matcher::fingerprint_file(write_file('tiny.txt', 'one two'), $K, $W), [], 'sub-k-gram file => empty'; +}; + +done_testing; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/Cavil-Matcher-1.01/typemap new/Cavil-Matcher-1.05/typemap --- old/Cavil-Matcher-1.01/typemap 2026-07-21 00:27:23.810104489 +0200 +++ new/Cavil-Matcher-1.05/typemap 2026-08-27 13:20:49.110595166 +0200 @@ -1,4 +1,6 @@ -Cavil::Matcher::Engine T_PTROBJ -Cavil::Matcher::Hash T_PTROBJ -Cavil::Matcher::Bag T_PTROBJ -AV* T_AVREF_REFCOUNT_FIXED +Cavil::Matcher::Engine T_PTROBJ +Cavil::Matcher::Hash T_PTROBJ +Cavil::Matcher::Bag T_PTROBJ +Cavil::Matcher::Fingerprints T_PTROBJ +AV* T_AVREF_REFCOUNT_FIXED +HV* T_HVREF_REFCOUNT_FIXED ++++++ _scmsync.obsinfo ++++++ --- /var/tmp/diff_new_pack.X8xdmx/_old 2026-09-08 17:01:55.682526462 +0200 +++ /var/tmp/diff_new_pack.X8xdmx/_new 2026-09-08 17:01:55.689526754 +0200 @@ -1,6 +1,6 @@ -mtime: 1785916160 -commit: a3fe64e68ac939b73fcd0b0e07207129b78d41c9f326f32c4073c47d6e5167fc +mtime: 1788514856 +commit: a1e7d2f055cf1f959a04e70b4cf478cbb42d3cd7d741fb5f737cb1c408ed7911 url: https://src.opensuse.org/perl/perl-Cavil-Matcher -revision: a3fe64e68ac939b73fcd0b0e07207129b78d41c9f326f32c4073c47d6e5167fc +revision: a1e7d2f055cf1f959a04e70b4cf478cbb42d3cd7d741fb5f737cb1c408ed7911 projectscmsync: https://src.opensuse.org/perl/_ObsPrj ++++++ build.specials.obscpio ++++++ ++++++ build.specials.obscpio ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.gitignore new/.gitignore --- old/.gitignore 1970-01-01 01:00:00.000000000 +0100 +++ new/.gitignore 2026-09-04 11:40:56.000000000 +0200 @@ -0,0 +1,5 @@ +*.obscpio +*.osc +_build.* +_service:* +.pbuild
