Processed: Re: [checks/cruft] Microsoft help files can be built using free software

[Debian Bug Tracking System]

Processing commands for cont...@bugs.debian.org:

> tags 867673 + pending
Bug #867673 [lintian] [checks/cruft] Microsoft help files can be built using 
free software
Added tag(s) pending.
> thanks
Stopping processing here.

Please contact me if you need assistance.
-- 
867673: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867673
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#867673: [checks/cruft] Microsoft help files can be built using free software

[Chris Lamb]

tags 867673 + pending
thanks

Actually, this wasn't too ugly in the end:

  
https://anonscm.debian.org/git/lintian/lintian.git/commit/?id=1371b6b57d571459a4a9ec7481ea2e73aea283cd


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb, Debian Project Leader
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

[lintian] 01/01: Don't emit source-contains-prebuilt-ms-help-file for files generated by Halibut. (Closes: #867673)

[Chris Lamb]

This is an automated email from the git hooks/post-receive script.

lamby pushed a commit to branch master
in repository lintian.

commit 1371b6b57d571459a4a9ec7481ea2e73aea283cd
Author: Chris Lamb 
Date:   Sun Jul 9 23:19:19 2017 +0100

Don't emit source-contains-prebuilt-ms-help-file for files generated by 
Halibut. (Closes: #867673)
---
 checks/cruft.pm | 7 +++
 data/cruft/warn-file-type   | 2 --
 debian/changelog| 2 ++
 t/tests/cruft-general-upstream/pre_upstream | 2 ++
 4 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/checks/cruft.pm b/checks/cruft.pm
index a008406..a9d72c6 100644
--- a/checks/cruft.pm
+++ b/checks/cruft.pm
@@ -643,6 +643,13 @@ sub find_cruft {
 close($fd);
 }
 
+# .chm files are usually generated by non-free software
+if (   $basename =~ /\.chm$/i
+&& $entry->is_open_ok
+&& index($entry->file_contents, 'Halibut,') == -1) {
+tag 'source-contains-prebuilt-ms-help-file', $name;
+}
+
 # Lena Söderberg image
 if ($basename =~ /\blenn?a\b/i) {
 if($file_info =~ /\bimage\b/i
diff --git a/data/cruft/warn-file-type b/data/cruft/warn-file-type
index 6fcce0c..22b9c36 100644
--- a/data/cruft/warn-file-type
+++ b/data/cruft/warn-file-type
@@ -12,8 +12,6 @@ source-contains-prebuilt-flash-project  ~~ ^Composite \s 
Document \s File
 source-contains-prebuilt-java-object~~ (?: Java [ ] (?:Jar [ ] 
file|archive [ ] data) | Zip [ ] archive | JAR)   ~~ (?i)\.jar$
 # do not forget to change also $JS_EXT in file.pm
 source-contains-prebuilt-javascript-object  ~~ .*  
  ~~ 
(?i)[-._](?:compiled|compressed|lite|min|pack(?:ed)?|yc)\.js$ ~~ 
map{'s/(?i)(?:[-._](?:compiled|compressed|lite|min|pack(?:ed)?|prod|yc))?\.js$/'.$_.'/'}
 qw(.js _orig.js .js.orig .src.js -src.js .debug.js -debug.js -nc.js)
-# ok chm could be created from docbook or texi
-source-contains-prebuilt-ms-help-file   ~~ ^MS \s Windows \s Html \s? Help 
\s Data ~~  (?i)\.chm$
 source-contains-prebuilt-python-object  ~~ ^python \s \d(\.\d+)? \s 
byte-compiled~~ .*  
  ~~ s/(?i)(?:\.cpython-\d{2}|\.pypy)?\.py[co]$/.py/
 source-contains-prebuilt-silverlight-object ~~ ^Zip \s archive \s data 
  ~~ (?i)\.xac$
 source-contains-prebuilt-windows-binary ~~ 
\b(?:PE(?:32|64)|(?:MS-DOS|COFF)\s executable)\b
diff --git a/debian/changelog b/debian/changelog
index e13196c..0f8f64f 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -13,6 +13,8 @@ lintian (2.5.52) UNRELEASED; urgency=medium
   * checks/cruft.{desc,pm}:
 + [CL] Clarify language of source-contains-prebuilt-ms-help-file
   description; there exists free software to generate .chm files.
++ [CL] Don't emit source-contains-prebuilt-ms-help-file for files
+  generated by Halibut. (Closes: #867673)
   * checks/fields.{desc,pm}:
 + [NT] Remove check for missing versioned build-depends for dpkg
   and debhlper when using Build-Profiles.  The necessary versions
diff --git a/t/tests/cruft-general-upstream/pre_upstream 
b/t/tests/cruft-general-upstream/pre_upstream
index 693d5a9..a60ffbe 100755
--- a/t/tests/cruft-general-upstream/pre_upstream
+++ b/t/tests/cruft-general-upstream/pre_upstream
@@ -37,6 +37,8 @@ echo 'configure-generated-file-in-source' > 
"${dir}/config.cache"
 printf 
"MZ\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x3c"
 > "${dir}/fake-win32-bin.exe"
 # use old magic \001\000\000\000 part is optional but needed with old file
 printf "ITSF\003\000\000\000\140\000\000\000\001\000\000\000" > 
"${dir}/fake-help-file.chm"
+# .chm files with "Halibut" are probably generated by free software
+printf "ITSF\003\000\000\000\140\000\000\000\001\000\000\000Halibut," > 
"${dir}/fake-help-file-halibut.chm"
 
 printf "CWS\011x" > "${dir}/fakeflash.swf"
 echo '//comment' > "${dir}/fake.min.js"

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] branch master updated (c532f80 -> 1371b6b)

[Chris Lamb]

This is an automated email from the git hooks/post-receive script.

lamby pushed a change to branch master
in repository lintian.

  from  c532f80   c/cruft.{desc,pm}: Clarify language of 
source-contains-prebuilt-ms-help-file description; there exists free software 
to generate .chm files.
   new  1371b6b   Don't emit source-contains-prebuilt-ms-help-file for 
files generated by Halibut. (Closes: #867673)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "adds" were already present in the repository and have only
been added to this reference.


Summary of changes:
 checks/cruft.pm | 7 +++
 data/cruft/warn-file-type   | 2 --
 debian/changelog| 2 ++
 t/tests/cruft-general-upstream/pre_upstream | 2 ++
 4 files changed, 11 insertions(+), 2 deletions(-)

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

Bug#867673: [checks/cruft] Microsoft help files can be built using free software

[Chris Lamb]

Hi,

>  * Lintian currently says that "There is Linux software to read them and
>  an incomplete FreePascal related project to create them", but Halibut
>  >= 1.2 can create them, so the implication that there is no complete
>  free software to create them should be corrected.

I've changed that here:

  
https://anonscm.debian.org/git/lintian/lintian.git/commit/?id=c532f80594b387f7e26d30cf7246055b07e4ef50

> * The cruft check should exclude files generated by such free software.
>   As a first pass I'd suggest excluding files that contain the string
>   "Halibut,".

… but unfortunately I can't see a tidy way of doing this using the
./data/cruft/warn-file-type machinery.

Any ideas?

My quick-and-dirty testcase is:

  --- a/t/tests/cruft-general-upstream/pre_upstream
  +++ b/t/tests/cruft-general-upstream/pre_upstream
  @@ -38,6 +38,9 @@ printf 
  "MZ\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
   # use old magic \001\000\000\000 part is optional but needed with old file
   printf "ITSF\003\000\000\000\140\000\000\000\001\000\000\000" > 
"${dir}/fake-help-file.chm"

  +# .chm files with "Halibut" are probably generated by free software
  +printf "ITSF\003\000\000\000\140\000\000\000\001\000\000\000Halibut," > 
"${dir}/fake-help-file-halibut.chm"
  +
   printf "CWS\011x" > "${dir}/fakeflash.swf"
   echo '//comment' > "${dir}/fake.min.js"
   # Make python generate python bytecode files


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb, Debian Project Leader
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

[lintian] 01/01: c/cruft.{desc,pm}: Clarify language of source-contains-prebuilt-ms-help-file description; there exists free software to generate .chm files.

[Chris Lamb]

This is an automated email from the git hooks/post-receive script.

lamby pushed a commit to branch master
in repository lintian.

commit c532f80594b387f7e26d30cf7246055b07e4ef50
Author: Chris Lamb 
Date:   Sun Jul 9 22:54:19 2017 +0100

c/cruft.{desc,pm}: Clarify language of 
source-contains-prebuilt-ms-help-file description; there exists free software 
to generate .chm files.
---
 checks/cruft.desc | 5 ++---
 debian/changelog  | 3 +++
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/checks/cruft.desc b/checks/cruft.desc
index e0da7c7..31a3d31 100644
--- a/checks/cruft.desc
+++ b/checks/cruft.desc
@@ -472,9 +472,8 @@ Info: The source tarball contains a prebuilt Microsoft 
precompiled help
  CHM files are mainly produced by proprietary, Windows-specific software.
  They are also mainly consumed by the Microsoft HTML Help Workshop.
  .
- There is Linux software to read them and an incomplete
- FreePascal related project to create them,
- but any examples in source packages are likely to be created
+ Whilst there is free software to read and write them, any
+ examples existing in source packages are likely to be created
  by the proprietary Microsoft software and are probably missing
  the source HTML and associated files.
  .
diff --git a/debian/changelog b/debian/changelog
index 20088a6..e13196c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -10,6 +10,9 @@ lintian (2.5.52) UNRELEASED; urgency=medium
 + [NT] Remove check for missing versioned build-depends for dpkg
   and debhlper when using Build-Profiles.  The necessary versions
   are now in oldstable.
+  * checks/cruft.{desc,pm}:
++ [CL] Clarify language of source-contains-prebuilt-ms-help-file
+  description; there exists free software to generate .chm files.
   * checks/fields.{desc,pm}:
 + [NT] Remove check for missing versioned build-depends for dpkg
   and debhlper when using Build-Profiles.  The necessary versions

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] branch master updated (d215014 -> c532f80)

[Chris Lamb]

This is an automated email from the git hooks/post-receive script.

lamby pushed a change to branch master
in repository lintian.

  from  d215014   d/changelog: Document reporting changes
   new  c532f80   c/cruft.{desc,pm}: Clarify language of 
source-contains-prebuilt-ms-help-file description; there exists free software 
to generate .chm files.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "adds" were already present in the repository and have only
been added to this reference.


Summary of changes:
 checks/cruft.desc | 5 ++---
 debian/changelog  | 3 +++
 2 files changed, 5 insertions(+), 3 deletions(-)

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] branch master updated (aff1b40 -> d215014)

[Niels Thykier]

This is an automated email from the git hooks/post-receive script.

nthykier pushed a change to branch master
in repository lintian.

  from  aff1b40   spelling: Add another correction
   new  003f95e   c/r-html-reports: Scope a map to reuse memory
   new  cf82f90   c/r-html-reports: Save two hash lookups in map_maintainer
   new  d1e6aa0   c/r-sync-state: Use b64 encoded checksums
   new  d215014   d/changelog: Document reporting changes

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "adds" were already present in the repository and have only
been added to this reference.


Summary of changes:
 commands/reporting-html-reports.pm | 32 
 commands/reporting-sync-state.pm   |  9 ++---
 debian/changelog   |  8 
 3 files changed, 26 insertions(+), 23 deletions(-)

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] 03/04: c/r-sync-state: Use b64 encoded checksums

[Niels Thykier]

This is an automated email from the git hooks/post-receive script.

nthykier pushed a commit to branch master
in repository lintian.

commit d1e6aa09649aba10bcc606b1355f1e1356c74aae
Author: Niels Thykier 
Date:   Fri Jul 7 08:24:08 2017 +

c/r-sync-state: Use b64 encoded checksums

Reduces the size of the checksums fields (sha256) by ~30%.  These
fields are unconditionally loaded even for reporting-html-reports that
do not actually use them for anything.

Signed-off-by: Niels Thykier 
---
 commands/reporting-sync-state.pm | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/commands/reporting-sync-state.pm b/commands/reporting-sync-state.pm
index 7bd9b27..20f0c82 100644
--- a/commands/reporting-sync-state.pm
+++ b/commands/reporting-sync-state.pm
@@ -25,6 +25,7 @@ use autodie;
 use Getopt::Long();
 use File::Basename qw(basename);
 use YAML::XS ();
+use MIME::Base64 qw(encode_base64);
 
 use Lintian::Lab;
 use Lintian::Relation::Version qw(versions_comparator);
@@ -334,8 +335,9 @@ sub _parse_srcs_pg {
 strip($f);
 next unless $f && $f =~ m/\.dsc$/;
 my ($checksum, undef, $basename) = split(m/\s++/, $f);
+my $b64_checksum = encode_base64(pack('H*', $checksum));
 # $dir should end with a slash if it is non-empty.
-$data{$DEFAULT_CHECKSUM} = $checksum;
+$data{$DEFAULT_CHECKSUM} = $b64_checksum;
 $data{'path'} = $extra_metadata->{'mirror-dir'}  . "/$dir" . $basename;
 last;
 }
@@ -356,7 +358,7 @@ sub _parse_srcs_pg {
 # Helper for local_mirror_manifests - it parses a paragraph from Packages file
 sub _parse_pkgs_pg {
 my ($state, $extra_metadata, $type, $paragraph) = @_;
-my ($group_id, $member_id, %data, %group_metadata);
+my ($group_id, $member_id, %data, %group_metadata, $b64_checksum);
 my $package = $paragraph->{'package'};
 my $version = $paragraph->{'version'};
 my $architecture = $paragraph->{'architecture'};
@@ -373,7 +375,8 @@ sub _parse_pkgs_pg {
 $member_id = "${type}:${package}/${version}/${architecture}";
 $data{'path'}
   = $extra_metadata->{'mirror-dir'} . '/' . $paragraph->{'filename'};
-$data{$DEFAULT_CHECKSUM} = $paragraph->{$DEFAULT_CHECKSUM};
+$b64_checksum = encode_base64(pack('H*', $paragraph->{$DEFAULT_CHECKSUM}));
+$data{$DEFAULT_CHECKSUM} = $b64_checksum;
 
 $group_metadata{'mirror-metadata'}{'maintainer'}
   = $paragraph->{'maintainer'};

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] 02/04: c/r-html-reports: Save two hash lookups in map_maintainer

[Niels Thykier]

This is an automated email from the git hooks/post-receive script.

nthykier pushed a commit to branch master
in repository lintian.

commit cf82f90f0d97cb98aad98bb8efe345f71a9d7036
Author: Niels Thykier 
Date:   Thu Jul 6 20:33:11 2017 +

c/r-html-reports: Save two hash lookups in map_maintainer

Signed-off-by: Niels Thykier 
---
 commands/reporting-html-reports.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/commands/reporting-html-reports.pm 
b/commands/reporting-html-reports.pm
index 5cf2e2e..f66874f 100644
--- a/commands/reporting-html-reports.pm
+++ b/commands/reporting-html-reports.pm
@@ -1019,8 +1019,8 @@ sub parse_lintian_log {
 sub map_maintainer {
 my ($urlmap, $maintainer) = @_;
 my $url = maintainer_url($maintainer);
-if ($urlmap->{$url} && $urlmap->{$url} ne $maintainer) {
-$maintainer = $urlmap->{$url};
+if (defined(my $res = $urlmap->{$url})) {
+$maintainer = $res;
 } else {
 $urlmap->{$url} = $maintainer;
 }

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] 01/04: c/r-html-reports: Scope a map to reuse memory

[Niels Thykier]

This is an automated email from the git hooks/post-receive script.

nthykier pushed a commit to branch master
in repository lintian.

commit 003f95ea74542ebbb33483b7353e4cdf317e6917
Author: Niels Thykier 
Date:   Thu Jul 6 20:31:26 2017 +

c/r-html-reports: Scope a map to reuse memory

Signed-off-by: Niels Thykier 
---
 commands/reporting-html-reports.pm | 32 
 1 file changed, 12 insertions(+), 20 deletions(-)

diff --git a/commands/reporting-html-reports.pm 
b/commands/reporting-html-reports.pm
index 4d663e2..5cf2e2e 100644
--- a/commands/reporting-html-reports.pm
+++ b/commands/reporting-html-reports.pm
@@ -820,7 +820,7 @@ sub parse_lintian_log {
 # The "last_*" are optimizations to avoid computing the same
 # things over and over again when a package have multiple tags.
 my (%seen, $last_info, $last_maintainer, %unknown_member_id, $info,
-$last_pi);
+$last_pi, %map_maint);
 my %expanded_code = (
 E => 'errors',
 W => 'warnings',
@@ -953,7 +953,8 @@ sub parse_lintian_log {
 # and, if so, map them to the same person as the previous
 # one.
 
-$last_maintainer = $maintainer = map_maintainer($maintainer);
+$last_maintainer = $maintainer
+  = map_maintainer(\%map_maint, $maintainer);
 
 # Update maintainer statistics.
 $statistics{maintainers}++ unless defined $by_maint{$maintainer};
@@ -995,7 +996,7 @@ sub parse_lintian_log {
 # d/control for some uploaders.
 $uploader =~ s/^"(.*)" <(.*)>$/$1 <$2>/;
 }
-$uploader = map_maintainer($uploader);
+$uploader = map_maintainer(\%map_maint, $uploader);
 next if $uploader eq $maintainer;
 $by_uploader{$uploader}{$source}{$source_version}
   = $list_ref;
@@ -1015,24 +1016,15 @@ sub parse_lintian_log {
 # Deduplicate maintainers.  Maintains a cache of the maintainers we've seen
 # with a given e-mail address and returns the maintainer string that we
 # should use (which is whatever maintainer we saw first with that e-mail).
-
-# PerlCritic apparently does not recognise this way of "encapsulating"
-# variables.  Though, these could probably be replaced by "state"
-# variables.
-## no critic (ControlStructures::ProhibitUnreachableCode)
-{
-my (%urlmap);
-
-sub map_maintainer {
-my ($maintainer) = @_;
-my $url = maintainer_url($maintainer);
-if ($urlmap{$url} && $urlmap{$url} ne $maintainer) {
-$maintainer = $urlmap{$url};
-} else {
-$urlmap{$url} = $maintainer;
-}
-return $maintainer;
+sub map_maintainer {
+my ($urlmap, $maintainer) = @_;
+my $url = maintainer_url($maintainer);
+if ($urlmap->{$url} && $urlmap->{$url} ne $maintainer) {
+$maintainer = $urlmap->{$url};
+} else {
+$urlmap->{$url} = $maintainer;
 }
+return $maintainer;
 }
 
 # Quote special characters for HTML output.

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] 04/04: d/changelog: Document reporting changes

[Niels Thykier]

This is an automated email from the git hooks/post-receive script.

nthykier pushed a commit to branch master
in repository lintian.

commit d2150144d0487f8ab460c0f73258b498241766ba
Author: Niels Thykier 
Date:   Sun Jul 9 20:17:34 2017 +

d/changelog: Document reporting changes

Signed-off-by: Niels Thykier 
---
 debian/changelog | 8 
 1 file changed, 8 insertions(+)

diff --git a/debian/changelog b/debian/changelog
index fad80ef..20088a6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -31,6 +31,14 @@ lintian (2.5.52) UNRELEASED; urgency=medium
 
   * commands/lintian.pm:
 + [NT] Attempt to clean up on SIGTERM (like with SIGINT).
+  * commands/reporting-html-reports:
++ [NT] Enable re-use of the memory for the maintainer mapping once the
+  log file as been parsed.
+  * commands/reporting-sync-state:
++ [NT] Re-encode checksums into base64 encoded strings (from hex).
+  This reduces size per checksum by about ~30% (disk and memory).  This
+  reduction is also applicable to other reporting tools that loads the
+  harness cache.
 
   * data/fields/build-profiles:
 + [NT] Add patch from Mattia Rizzolo to update the list of known build

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] branch master updated (887c381 -> aff1b40)

[Paul Wise]

This is an automated email from the git hooks/post-receive script.

pabs pushed a change to branch master
in repository lintian.

  from  887c381   t: Expand existing test a bit
   new  aff1b40   spelling: Add another correction

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "adds" were already present in the repository and have only
been added to this reference.


Summary of changes:
 data/spelling/corrections | 1 +
 1 file changed, 1 insertion(+)

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git

[lintian] 01/01: spelling: Add another correction

[Paul Wise]

This is an automated email from the git hooks/post-receive script.

pabs pushed a commit to branch master
in repository lintian.

commit aff1b40f696806c056e658aef2539aebbc6cbffb
Author: Paul Wise 
Date:   Mon Jul 10 00:28:13 2017 +1000

spelling: Add another correction
---
 data/spelling/corrections | 1 +
 1 file changed, 1 insertion(+)

diff --git a/data/spelling/corrections b/data/spelling/corrections
index afa421a..384f33e 100644
--- a/data/spelling/corrections
+++ b/data/spelling/corrections
@@ -1011,6 +1011,7 @@ debain||Debian
 debiab||Debian
 debia||Debian
 debians||Debian's
+debugginf||debugging
 debuging||debugging
 decalared||declared
 decalare||declare

-- 
Alioth's /usr/local/bin/git-commit-notice on 
/srv/git.debian.org/git/lintian/lintian.git