[Reproducible-builds] Bug#806891: Multi-file HTML output

[Joachim Breitner]

Package: diffoscope
Version: 42
Severity: wishlist
Tags: patch

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

in the branch "html-dir" in the main repository, you will find the
following feature:

Multi-file HTML output

in order to make it feasible to create HTML output with large diffs,
this adds a new presenter mode (--html-dir) which spreads the output
over multiple files in a directory.

In particular, the table presenting a diff that is larger than the
number of bytes specified via --separate-file-diff-size (200kB by
default) is writen to a separate file and loaded on demand using
JavaScript/JQuery.

By default, JQuery is symlinked into the output directory from
/usr/share/javascript/jquery/jquery.js (install libjs-jquery!), but an
alternative location (e.g. /javascript/jquery/jquery.js) can be
specified using the --jquery command line parameter.

http://anonscm.debian.org/cgit/reproducible/diffoscope.git/log/?h=html-dir

Please review, and if you like it, merge it and remove the html-dir branch.

Greetings,
Joachim


- -- System Information:
Debian Release: stretch/sid
  APT prefers buildd-unstable
  APT policy: (500, 'buildd-unstable'), (500, 'unstable'), (101, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python3-libarchive-c   2.1-3
ii  python3-magic  1:5.25-2
ii  python3-pkg-resources  18.4-2
pn  python3:any

Versions of packages diffoscope recommends:
ii  acl2.2.52-2
pn  binutils-multiarch 
ii  bzip2  1.0.6-8
ii  cpio   2.11+dfsg-4.1
ii  default-jdk [java-sdk] 2:1.7-52.1
ii  fontforge-extras   0.3-4
ii  fp-utils   2.6.4+dfsg-8
ii  fp-utils-2.6.4 [fp-utils]  2.6.4+dfsg-8
ii  gcj-4.8-jdk [java-sdk] 4.8.5-2
ii  gcj-4.9-jdk [java-sdk] 4.9.3-7
ii  gcj-5-jdk [java-sdk]   5.2.1-23
ii  gcj-jdk [java-sdk] 4:5.2.1-6
ii  genisoimage9:1.1.11-3
ii  gettext0.19.6-1
ii  ghc7.8.4-9
ii  gnupg  1.4.19-6
ii  mono-utils 3.2.8+dfsg-10
ii  openjdk-6-jdk [java-sdk]   6b37-1.13.9-1
ii  openjdk-7-jdk [java-sdk]   7u91-2.6.3-2
ii  pdftk  2.02-3
ii  poppler-utils  0.26.5-4
ii  python3-debian 0.1.27
ii  python3-rpm4.12.0.1+dfsg1-3+b1
ii  python3-tlsh   3.4.1+20151014-1
ii  rpm2cpio   4.12.0.1+dfsg1-3+b1
ii  sng1.0.6-2
ii  sqlite33.9.2-1
ii  squashfs-tools 1:4.3-3
ii  unzip  6.0-20
ii  vim-common 2:7.4.826-1+b1
ii  xz-utils   5.1.1alpha+20120614-2.1

diffoscope suggests no packages.

- -- no debconf information

-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlZfCmIACgkQ9ijrk0dDIGxJRgCfZzj7cMjbL+3kTBMs/fTjUcer
9Q8An0cIY9ylDdmHNKKQ818UdXAFgliN
=YIRi
-END PGP SIGNATURE-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806891: Bug#806891: Multi-file HTML output

[Holger Levsen]

Hi Joachim,

On Mittwoch, 2. Dezember 2015, Joachim Breitner wrote:
> in the branch "html-dir" in the main repository, you will find the
> following feature:
> Multi-file HTML output

/me likes, thanks for implementing this!
 
> In particular, the table presenting a diff that is larger than the
> number of bytes specified via --separate-file-diff-size (200kB by
> default) is writen to a separate file and loaded on demand using
> JavaScript/JQuery.

just using javascript for this I dislike, I'm very happy that 
reproducible.debian.net doesnt need javascript for any functionality at all 
(and I would like to use multi-html output there too), so how about adding 
another option to create static links (or whatever non-JS solution) instead.


cheers,
Holger


signature.asc
Description: This is a digitally signed message part.
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806891: [Reproducible-commits] [diffoscope] 01/01: Multi-file HTML output

[Jérémy Bobbio]

Joachim Breitner:
> Multi-file HTML output

Really great idea. :) Thanks for the initial patch!

> +parser.add_argument('--jquery', metavar='url', dest='jquery_url',
> +help='link to the jquery url, with --html-dir. By 
> default, a symlink to /usr/share/javascript/jquery/jquery.js is created')

To make the Suggests a reality, I think it would be better to only
display “By default, a symlink…” if the file is already present on the
filesystem. Otherwise, if `--html-dir` is specified, the software should
exit with an error, asking users to specify `--jquery`.

Related question: is the browser going to visit the subpage if I have
JavaScript disabled? In that case, one other option is to only add the
JavaScript code when we have access to jQuery. I know some people don't
want JavaScript in their browser, so specifying something like
`--no-jquery` or `--jquery=none` could also be a way to turn it off.

> +# no output desired? print text
> +if not parsed_args.text_output and not parsed_args.html_output and 
> not parsed_args.html_output_directory:
> +parsed_args.text_output = "-"

Maybe it would be nicer to write:

if not any(parsed_args.text_output, parsed_args.html_output, 
parsed_args.html_output_directory):

> +def output_unified_diff(print_func, directory, anchor, unified_diff):
> +if directory and len(unified_diff) > 
> Config.general.separate_file_diff_size:
> +# open a new file for this table
> +filename="%s.html" % hashlib.md5(anchor.encode('utf-8')).hexdigest()

I'm not entirely sure the anchor as it's working right now will be
unique… 

> +logger.debug('separate html output for diff of %s (size %d)', 
> anchor, len(unified_diff))
> +with file_printer(directory, filename) as new_print_func:
> +output_unified_diff_table(new_print_func, unified_diff)

So I think it would be great to crash here instead of overwrite if the file
aleardy exists. What do you think?

-- 
Lunar.''`. 
lu...@debian.org: :Ⓐ  :  # apt-get install anarchism
`. `'` 
  `-   


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806891: [Reproducible-commits] [diffoscope] 01/01: Multi-file HTML output

[Joachim Breitner]

Hi,

Am Mittwoch, den 02.12.2015, 17:32 +0200 schrieb Jérémy Bobbio:

> 
> > +parser.add_argument('--jquery', metavar='url', dest='jquery_url',
> > +help='link to the jquery url, with --html-dir. By 
> > default, a symlink to /usr/share/javascript/jquery/jquery.js is created')
> 
> To make the Suggests a reality, I think it would be better to only
> display “By default, a symlink…” if the file is already present on the
> filesystem. Otherwise, if `--html-dir` is specified, the software should
> exit with an error, asking users to specify `--jquery`.

You want --help output that depends on the system state?

> Related question: is the browser going to visit the subpage if I have
> JavaScript disabled? In that case, one other option is to only add the
> JavaScript code when we have access to jQuery. I know some people don't
> want JavaScript in their browser, so specifying something like
> `--no-jquery` or `--jquery=none` could also be a way to turn it off.

See my other mail, I can make it degrade gracefully. Those people will
have JavaScript disabled, so I don’t see a strong usecase for multi-
page html output with JavaScript completely disabled. Note that --html
is an option for those users.

> > +# no output desired? print text
> > +if not parsed_args.text_output and not
> > parsed_args.html_output and not parsed_args.html_output_directory:
> > +parsed_args.text_output = "-"
> 
> Maybe it would be nicer to write:
> 
> if not any(parsed_args.text_output, parsed_args.html_output,
> parsed_args.html_output_directory):

✓

> > +def output_unified_diff(print_func, directory, anchor,
> > unified_diff):
> > +if directory and len(unified_diff) >
> > Config.general.separate_file_diff_size:
> > +# open a new file for this table
> > +filename="%s.html" % hashlib.md5(anchor.encode('utf-
> > 8')).hexdigest()
> 
> I'm not entirely sure the anchor as it's working right now will be
> unique… 

Ok, I’ll ditch this and simply hash the unified_diff: There, clashes
will not matter.

> > +logger.debug('separate html output for diff of %s (size
> > %d)', anchor, len(unified_diff))
> > +with file_printer(directory, filename) as new_print_func:
> > +output_unified_diff_table(new_print_func,
> > unified_diff)
> 
> So I think it would be great to crash here instead of overwrite if the file
> aleardy exists. What do you think?

Not with the above change.

Also, I want to be able to re-run diffoscope without having to manually
delete the output directory first.

Greetings,
Joachim

-- 
Joachim "nomeata" Breitner
Debian Developer
  nome...@debian.org | ICQ# 74513189 | GPG-Keyid: F0FBF51F
  JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806891: Bug#806891: Multi-file HTML output

[Joachim Breitner]

Hi,

Am Mittwoch, den 02.12.2015, 17:31 +0200 schrieb Holger Levsen:
> On Mittwoch, 2. Dezember 2015, Joachim Breitner wrote:
> > In particular, the table presenting a diff that is larger than the
> > number of bytes specified via --separate-file-diff-size (200kB by
> > default) is writen to a separate file and loaded on demand using
> > JavaScript/JQuery.
> 
> just using javascript for this I dislike, I'm very happy that 
> reproducible.debian.net doesnt need javascript for any functionality at all 
> (and I would like to use multi-html output there too), so how about adding 
> another option to create static links (or whatever non-JS solution) instead.

the buttons to load the diffs are actually links to the HTML of the
table, so it almost degrades gracefully.

To make it degrade gracefully fully, I would have to add a proper HTML
header around the table, and ignore that when inserting the fragment
dynamically. Both is easily possible. Is that ok then?

Geetings,
Joachim

-- 
Joachim "nomeata" Breitner
Debian Developer
  nome...@debian.org | ICQ# 74513189 | GPG-Keyid: F0FBF51F
  JID: nome...@joachim-breitner.de | http://people.debian.org/~nomeata



signature.asc
Description: This is a digitally signed message part
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] Second build on failures

[Vagrant Cascadian]

On 2015-12-01, Reiner Herrmann  wrote:
> On Tue, Dec 01, 2015 at 02:13:07PM -0800, Vagrant Cascadian wrote:
>> Hey, I think all of the second builds on armhf are failing to set up the
>> build environment:
>> 
>>   
>> https://reproducible.debian.net/logs/unstable/armhf/gb_0.3.2-1.build2.log.gz
>> 
>>   I: Installing the build-deps
>>   I: user script 
>> /srv/workspace/pbuilder/5651/tmp/hooks/D01_modify_environment starting
>>   FATAL: kernel too old
>
> Interesting... According to codesearch this comes from glibc [1].
> It could be related to "linux64 --uname-2.6", which we use use to fake a 
> different
> kernel version.
>
> [1]: 
> https://sources.debian.net/src/glibc/2.19-18/sysdeps/unix/sysv/linux/dl-osinfo.h/?hl=45#L45

Indeed, the "linux64 --uname-2.6" seems to be the culprit.

I first tested removing the linux64 call on two of the nodes, and they
were the only ones successfully running second builds for the last
several hours... so I've just now removed the linux64 calls on all the
armhf nodes.

Would be best to investigate the issue further...

On a related note, it might be worth trying "setarch uname26" instead of
linux64 (which is just a symlink to setarch). But I doubt if this will
change anything with the above issue.

live well,
  vagrant


signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806911: libc-bin: ldconfig segfaults when run using "setarch uname26"

[Vagrant Cascadian]

Package: libc-bin
Version: 2.21-1
Severity: normal
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Apparently, when run with "setarch uname26" or "linux64 --uname-2.6",
ldconfig segfaults.

  setarch uname26 ldconfig
  FATAL: kernel too old
  Segmentation fault

libc-bin version 2.19-22 in stretch does not segfault when run this
way.

I haven't tried, but this may also fail similarly when run on an old
kernel as well.

At the very least, maybe it shouldn't segfault with old kernels.

The reproducible builds project use "linux64 --uname-2.6" to set a
different kernel version for the second build to find bugs in packages
that build differently depending on the running kernel version, and it
would be nice if this would continue to work.


live well,
  vagrant


signature.asc
Description: PGP signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] Bug#806859: libnet-frame-perl: FTBFS: Attempts to access internet during build

[Chris Lamb]

Source: libnet-frame-perl
Version: 1.16-1
Severity: serious
Justification: fails to build from source
User: reproducible-builds@lists.alioth.debian.org
Usertags: ftbfs
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org

Dear Maintainer,

libnet-frame-perl fails to build from source in unstable/amd64:

  [..]

  PPP: address:0xff  control:0x03  protocol:0x0021
  [IPv4]
  ok 1
  ok
  Net::Frame::Layer::getHostIpv4Addr: unable to resolv `gomor.org'
  hostname
   at t/13-gethostsubs.t line 15.
  # Test 1 got: "0" (t/13-gethostsubs.t at line 23)
  #   Expected: "1"
  #  t/13-gethostsubs.t line 23 is:1,
  Net::Frame::Layer::__ANON__: unable to resolv `gomor.org' hostname:
  getaddrinfo: Temporary failure in name resolution
   at t/13-gethostsubs.t line 60.

  [..]

The full build log is attached or can be viewed here:


https://reproducible.debian.net/logs/unstable/amd64/libnet-frame-perl_1.16-1.build1.log.gz


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-


libnet-frame-perl.1.16-1.unstable.amd64.log.txt.gz
Description: Binary data
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

[Reproducible-builds] diffoscope dedicated mailing-list

[Jérémy Bobbio]

Hi!

diffoscope is really growing bigger than just the Debian community, so
it felt better to create a dedicated mailing-list.

Please subscribe: https://lists.reproducible-builds.org/listinfo/diffoscope

-- 
Lunar.''`. 
lu...@debian.org: :Ⓐ  :  # apt-get install anarchism
`. `'` 
  `-   


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds