Re: [Reproducible-builds] Bug#826711: diffoscope: please add argument completion
Hi, On Sun, Jun 19, 2016 at 8:16 AM, Paul Wise wrote: > On Sun, 2016-06-19 at 00:32 +0530, Satyam Zode wrote: > >> I made the changes as you mentioned, please find an attached patches. > > Personally, I would suggest that the changes you have separated into > separate patches per file are a single logical change and as such > should all be made in the same commit. Some discussion of this "logical > change" concept from the OpenStack community is here: > > https://wiki.openstack.org/wiki/GitCommitMessages > Thanks :) Good resource for learning! >> I have removed this temporarily! we already had discussion regarding >> this on IRC. Looking forward to your response! > > I would suggest leaving the range completers in, even with the enforced > sorting by bash itself, I think they are useful. For the upper end of > the completion range, I think go with double the current maximum. Here is range completion using updated patch: satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope - --css --help --max-diff-block-lines --text --debug--html --max-diff-input-lines --version --debugger --html-dir --max-report-size --fuzzy-threshold --jquery --new-file -h --list-tools --separate-file-diff-size satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope --max-report-size 0100 120 140 160 180 20 200 40 60 80 satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope --max-diff-block-lines 0 10 15 20 25 30 35 40 45 5 50 satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope --max-diff-input-lines 0 10 2 3 4 500055000 65000 75000 85000 95000 1 15000 25000 35000 45000 5 6 7 8 9 satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope --separate-file-diff-size 0 10 12 14 16 18 2 20 4 6 8 satyam@satyamz:~/debian/satymaz-guest/copy/diffoscope/bin$ ./diffoscope --fuzzy-threshold 0100 120 140 160 180 20 200 220 240 260 280 300 320 340 360 380 40 400 60 80 > > Some further comments on the patches below: > >> +dh_auto_build: >> + debian/diffoscope.bash-completion >> + debian/diffoscope.1 > > This will cause the build to fail, those three lines need to all be on > one single line with spaces in between. Did gmail wrap it for you? > > dh_auto_build: debian/diffoscope.bash-completion debian/diffoscope.1 > >> +++ b/debian/clean >> @@ -0,0 +1,3 @@ >> +rm -f debian/diffoscope.1 >> +dh_clean -O--buildsystem=pybuild > > The first two lines of the debian/clean file need fixes. See the manual > page for dh_clean, but the debian/clean file is simply a list of files > (one per line) to be removed by dh_clean from `debian/rules clean`. > The full debian/clean file for diffoscope should look like this: > > debian/diffoscope.1 > debian/diffoscope.bash-completion > >> +elif '_ARGCOMPLETE' not in os.environ: > > This will trigger the error in the circumstance where you don't have > argcomplete installed and you aren't asking for completion. This means > it will give an error when running it from the command-line if > argcomplete isn't installed. I think we want an error when you don't > have argcomplete installed but you are asking for completion. > To fix this, remove the "not" from this line. > >> +print('ERROR: Argument completion requested but Python argcomplete >> module not installed', file=sys.stderr) > > In another place in the diffoscope codebase, a critical error is > reported using logger.critical before exiting with an error. > I'm not sure if print, logger.critical or logger.error should be used > for this. I guess Lunar or other folks can advise you about this. > > logger.critical('Console is unable to print Unicode characters. Set e.g. > LC_CTYPE=en_US.UTF-8') > Here, I used logger.error for displaying error and if error occurs diffoscope returns. Thanks! Satyam Zode From 849db5cbf37905973e5171239e5222a673117536 Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Sun, 19 Jun 2016 15:14:42 +0530 Subject: [PATCH 2/2] Add dependencies for argument completion Add new rules for bash-completion script. Add new file debian/clean. --- debian/clean | 2 ++ debian/control | 2 ++ debian/rules | 13 +++-- 3 files changed, 11 insertions(+), 6 deletions(-) create mode 100644 debian/clean diff --git a/debian/clean b/debian/clean new file mode 100644 index 000..81040f5 --- /dev/nu
Re: [Reproducible-builds] Bug#826711: diffoscope: please add argument completion
Hi, all ! Pabs, Thanks for suggestions :) I made the changes as you mentioned, please find an attached patches. On Sun, Jun 12, 2016 at 1:31 PM, Paul Wise wrote: > On Sun, 2016-06-12 at 11:50 +0530, Satyam Zode wrote: > > > Tags: patch > > That only works in mails to sub...@bugs.debian.org, for mails to the > bug report address, you want this instead (-1 means the current bug): > > Control: tags -1 + patch > > > I have updated the patch as per suggestions please review it. > > Review below... > > > override_dh_clean: > > + debian/diffoscope.bash-completion > > This causes the package to fail to build. I think override_dh_clean > should be removed and both the bash completion script and > debian/diffoscope.1 added to debian/clean. Moving the existing things > to the debian/clean file should be done in a separate commit. > > > dh_python3 --recommends=python-debian --recommends=rpm-python > --recommends=tlsh --recommends=guestfs > > This line needs to add --recommends=argcomplete > > > +override_dh_auto_build: > > + register-python-argcomplete diffoscope > > debian/diffoscope.bash-completion > > This disables the upstream build system, you should call dh_auto_build > after calling register-python-argcomplete, I think this is a better way > to do it than what is there now: > > override_dh_auto_build: > debian/diffoscope.bash-completion debian/diffoscope.1 > > debian/diffoscope.bash-completion: > register-python-argcomplete diffoscope > $@ > I made above changes and here is build log : http://paste.debian.net/740041/ I am not sure if anything else is needed or not in debian/* files. > > You also need to add generated files to .gitignore so they don't get > accidentally committed to the repository. > > > +* ``python-argcomplete`` is used for argument completion. > > + Available on Debian and Fedora as > > + ``python-argcomplete``. > > + ``python-argcomplete`` is also available on `PyPI ` as > > + ``argcomplete``. > > I think I would write that like this: > > * ``argcomplete`` is used for argument completion. > Available on Debian as ``python3-argcomplete``. > Available on Fedora as ``python-argcomplete``. > Available on `PyPI <https://pypi.python.org/pypi/argcomplete/>`_. > > I have made these changes too. > > -default=Config.general.max_report_size) > > > + > default=Config.general.max_report_size).completer=RangeCompleter(0, > > +Config.general.max_report_size, 20) > > I think this prevents people from completing values above the > default max_report_size? Same for the other RangeCompleters. > > Interestingly, in the range completers, 100 appears to sort before > 80, so it is doing string sorting not numeric sorting. > I have removed this temporarily! we already had discussion regarding this on IRC. Looking forward to your response! Thanks! Satyam Zode From 43f3ed25ab3138cf353c3a34438cc7b8c8418e83 Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Sat, 18 Jun 2016 22:12:58 +0530 Subject: [PATCH 4/4] Added new rules for bash-completion script --- debian/rules | 15 +-- 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/debian/rules b/debian/rules index b15a73b..a4bdf79 100755 --- a/debian/rules +++ b/debian/rules @@ -8,10 +8,10 @@ ifneq ($(VERSION_dch),$(VERSION_py)) endif %: - dh $@ --with python3 --buildsystem=pybuild + dh $@ --with python3 --with bash-completion --buildsystem=pybuild override_dh_python3: - dh_python3 --recommends=python-debian --recommends=rpm-python --recommends=tlsh --recommends=guestfs + dh_python3 --recommends=python-debian --recommends=rpm-python --recommends=tlsh --recommends=guestfs --recommends=argcomplete override_dh_gencontrol: TOOLS="$$(bin/diffoscope --list-tools=debian | tail -n 1 | \ @@ -23,13 +23,16 @@ override_dh_gencontrol: debian/diffoscope.1: debian/diffoscope.1.rst rst2man $< $@ +debian/diffoscope.bash-completion: + register-python-argcomplete diffoscope > $@ + +dh_auto_build: + debian/diffoscope.bash-completion + debian/diffoscope.1 + override_dh_installman: debian/diffoscope.1 dh_installman -O--buildsystem=pybuild -override_dh_clean: - rm -f debian/diffoscope.1 - dh_clean -O--buildsystem=pybuild - diffoscope/presenters/icon.py: favicon.png (echo '# Generated from favicon.png'; \ echo 'FAVICON_BASE64 = """'; \ -- 2.1.4 From d7192fb92a4fca051bee2b13548878923c34995e Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Sat, 18 Jun 2016 22:05:19 +0530 Subject: [PATCH 3/4] Added debian/clean file --- debian/clean | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 debian/clean
Re: [Reproducible-builds] [diffoscope] Support for --ignore-profile flag in diffoscope
Hi, all ! I have been working on " hiding differences " options. I have tried to add some possible use cases, which I came across after researching different differences as suggested in above discussion,many thanks to all for their views. Here is wiki page https://wiki.debian.org/ReproducibleBuilds/HideProfilesSpecification Since, we have a couple of use cases. I think, it's a proper time to discuss and start thinking about implementation of these use cases as well as categorize the CLI interface for the options. Most importantly, we need to prioritize given use cases and decide which one to support and which one to ignore. Here, we also need to design these hiding options in such a way that user shall understand what he/she is hiding and how it'll affect the output. Please feel free to edit the wiki page to add new things. :) Looking forward to suggestions. Thanking you! Satyam Zode ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#826711: diffoscope: please add argument completion
Tags: patch Hi, Pabs! I have updated the patch as per suggestions please review it. Thanks! Satyam Zode, From a1389838a38aecd894e052635e4c3e73f3cf6ef4 Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Sun, 12 Jun 2016 11:46:02 +0530 Subject: [PATCH] Added support for argument completion * diffoscope/README.rst: Updated documentation for python-argcomplete * diffoscope/bin/diffoscope: Added env var PYTHON_ARGCOMPLETE_OK * diffoscope/debian/control: Added python-argcomplete as dependencies * diffoscope/debian/rules : Added bash-completion * diffoscope/__main__.py : Added completer class and argument completion support using argcomplete * Closes : #826711 --- README.rst | 5 + bin/diffoscope | 1 + debian/control | 2 ++ debian/rules | 6 +- diffoscope/__main__.py | 34 +- 5 files changed, 42 insertions(+), 6 deletions(-) diff --git a/README.rst b/README.rst index 5288e52..093a8ad 100644 --- a/README.rst +++ b/README.rst @@ -59,6 +59,11 @@ Optionally, the following modules will enhance it: ``python-magic``. It is built from `file <http://www.darwinsys.com/file/>`_. Available on Debian and Fedora as ``python3-magic``. +* ``python-argcomplete`` is used for argument completion. + Available on Debian and Fedora as + ``python-argcomplete``. + ``python-argcomplete`` is also available on `PyPI <https://pypi.python.org/pypi/argcomplete/>` as + ``argcomplete``. The various comparators rely on external commands being available. To get a list of them, please run:: diff --git a/bin/diffoscope b/bin/diffoscope index 2393807..2422b70 100755 --- a/bin/diffoscope +++ b/bin/diffoscope @@ -1,4 +1,5 @@ #!/usr/bin/env python3 +# PYTHON_ARGCOMPLETE_OK # -*- coding: utf-8 -*- # # diffoscope: in-depth comparison of files, archives, and directories diff --git a/debian/control b/debian/control index 19619d7..b24fbef 100644 --- a/debian/control +++ b/debian/control @@ -7,9 +7,11 @@ Uploaders: Mattia Rizzolo , Reiner Herrmann , Build-Depends: + bash-completion, binutils-multiarch, debhelper (>= 9), dh-python, + python-argcomplete, python3-all, python3-debian, python3-docutils, diff --git a/debian/rules b/debian/rules index b15a73b..5a4e75f 100755 --- a/debian/rules +++ b/debian/rules @@ -8,7 +8,7 @@ ifneq ($(VERSION_dch),$(VERSION_py)) endif %: - dh $@ --with python3 --buildsystem=pybuild + dh $@ --with python3 --with bash-completion --buildsystem=pybuild override_dh_python3: dh_python3 --recommends=python-debian --recommends=rpm-python --recommends=tlsh --recommends=guestfs @@ -27,9 +27,13 @@ override_dh_installman: debian/diffoscope.1 dh_installman -O--buildsystem=pybuild override_dh_clean: + debian/diffoscope.bash-completion rm -f debian/diffoscope.1 dh_clean -O--buildsystem=pybuild +override_dh_auto_build: + register-python-argcomplete diffoscope > debian/diffoscope.bash-completion + diffoscope/presenters/icon.py: favicon.png (echo '# Generated from favicon.png'; \ echo 'FAVICON_BASE64 = """'; \ diff --git a/diffoscope/__main__.py b/diffoscope/__main__.py index ac7913c..3306123 100644 --- a/diffoscope/__main__.py +++ b/diffoscope/__main__.py @@ -1,4 +1,5 @@ #!/usr/bin/env python3 +# PYTHON_ARGCOMPLETE_OK # -*- coding: utf-8 -*- # # diffoscope: in-depth comparison of files, archives, and directories @@ -30,6 +31,10 @@ try: import tlsh except ImportError: tlsh = None +try: +import argcomplete +except ImportError: +argcomplete = None from diffoscope import logger, VERSION, set_locale, clean_all_temp_files import diffoscope.comparators from diffoscope.config import Config @@ -60,25 +65,30 @@ def create_parser(): dest='max_report_size', type=int, help='maximum bytes written in report (default: %d)' % Config.general.max_report_size, -default=Config.general.max_report_size) +default=Config.general.max_report_size).completer=RangeCompleter(0, +Config.general.max_report_size, 20) parser.add_argument('--separate-file-diff-size', metavar='BYTES', dest='separate_file_diff_size', type=int, help='diff size to load diff on demand, with --html-dir (default: %d)' % Config.general.separate_file_diff_size, -default=Config.general.separate_file_diff_size) +default=Config.general.separate_file_diff_size).completer=RangeCompleter(0, + Config.general.separate_file_diff_size, 2) parser.add_argument('--max-diff-block-lines', dest='max_diff_block_lines', type=int, help='maximum numb
Re: [Reproducible-builds] Bug#826711: diffoscope: please add argument completion
Hi, all ! Paul Wise : > > Package: diffoscope > Severity: wishlist > Tags: newcomer > > Please add argument completion. Using the Python 3 argcomplete module > makes this relatively simple. This project can be used as an example: > > https://anonscm.debian.org/cgit/collab-maint/check-all-the-things.git/ Please find an attached patch which enables argument completion for diffoscope. Looking forward to review and feedback. Thanks! Satyam Zode From 50dffe772b38355eb673994fdea49d9d74be7cac Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Sat, 11 Jun 2016 16:26:28 +0530 Subject: [PATCH] Added support for argument completion. Closes #826711 --- bin/diffoscope | 1 + debian/control | 1 + diffoscope/__main__.py | 7 +++ 3 files changed, 9 insertions(+) diff --git a/bin/diffoscope b/bin/diffoscope index 2393807..2422b70 100755 --- a/bin/diffoscope +++ b/bin/diffoscope @@ -1,4 +1,5 @@ #!/usr/bin/env python3 +# PYTHON_ARGCOMPLETE_OK # -*- coding: utf-8 -*- # # diffoscope: in-depth comparison of files, archives, and directories diff --git a/debian/control b/debian/control index 19619d7..733d3be 100644 --- a/debian/control +++ b/debian/control @@ -20,6 +20,7 @@ Build-Depends: python3-rpm, python3-setuptools, python3-tlsh (>= 3.4.1), + python3-argcomplete, vim-common, Standards-Version: 3.9.8 Homepage: https://diffoscope.org/ diff --git a/diffoscope/__main__.py b/diffoscope/__main__.py index ac7913c..ab3739f 100644 --- a/diffoscope/__main__.py +++ b/diffoscope/__main__.py @@ -1,4 +1,5 @@ #!/usr/bin/env python3 +# PYTHON_ARGCOMPLETE_OK # -*- coding: utf-8 -*- # # diffoscope: in-depth comparison of files, archives, and directories @@ -30,6 +31,10 @@ try: import tlsh except ImportError: tlsh = None +try: +import argcomplete +except ImportError: +argcomplete = None from diffoscope import logger, VERSION, set_locale, clean_all_temp_files import diffoscope.comparators from diffoscope.config import Config @@ -89,6 +94,8 @@ def create_parser(): parser.add_argument('file2', help='second file to compare') if not tlsh: parser.epilog = 'File renaming detection based on fuzzy-matching is currently disabled. It can be enabled by installing the “tlsh” module available at https://github.com/trendmicro/tlsh' +if argcomplete: +argcomplete.autocomplete(parser) return parser -- 1.9.1 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Support for --ignore-profile flag in diffoscope
"Ximin Luo" : > For a start, there's the list of already-known issues. <https://tests.reproducible-builds.org/index_issues.html>https <https://tests.reproducible-builds.org/index_issues.html>:// <https://tests.reproducible-builds.org/index_issues.html> tests.reproducible-builds.org <https://tests.reproducible-builds.org/index_issues.html>/index_ <https://tests.reproducible-builds.org/index_issues.html>issues.html <https://tests.reproducible-builds.org/index_issues.html> and I'd imagine people analysing diffs would want an easy way to distinguish "issues that someone else has already solved" vs "issues nobody has seen before". Yes! I'm looking at those issues but it seems those issues are more in number. Frankly speaking, I'm not completely aware about many of those. Hence, I'm requesting you all to give some suggestions (and may be list of issues) so that I can start working on it further. Or maybe we can just categorize available issues :) > Concretely I have some suggestions: > > 1. instead of calling this "ignore" we call it "hide". and instead of "irrelevant" we say "common"/"minor"/"known" > > 2. diffoscope --ignore-* (or --hide-*) MUST NOT return 0 or otherwise give the impression that two non-identical files are the same, even if all differences are "hidden". It should report "n differences hidden". I'm agree with this terminology :) Thanks! Satyam Zode ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Support for --ignore-profile flag in diffoscope
Hi, all ! I am Satyam Zode, I am GSoC student intern (http://satyamz.github.io /blog/2016/05/08/google-summer-of-code-2016-with-debian -reproducible-builds-introduction/). I am trying to understand the problem "Allow users to ignore arbitrary differences" ( --ignore-profiles flag) in which diffoscope users will be able to ignore arbitrary differences. This problem is also described in diffoscope wishlist (https://reproducible-builds.org/events/athens2015/ diffoscope-wishlist/). I have started thinking about the solution of this problem and for that, I want to know as a diffoscope user, what kind of stuff would you like to ignore ? Such as any irrelevant differences which are just making noise and useless. Currently, I am looking at pkg-diff (provided by OBS) code because it's able to ignore some kind of stuff (Lunar has suggested me to have a look at pkg-diff to get an idea about this problem) . I kindly request you all to express your views and expectations regarding this particular problem :) Thanking you! *Satyam Zode* ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#818856: Bug#818856: diffoscope: crashes on broken symlinks
Hi Reiner! I did some trivial changes. Please find an attached patch. This patch closes #818856 and #796262. Thanks! Satyam Zode From 85c062d32b84ed6d8fcf5a80f0d2a2fbb4e43e54 Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Thu, 7 Apr 2016 01:15:04 +0530 Subject: [PATCH] Fixed issue related to diffoscope symlinks crashing (Closes: #818856,#796262) --- diffoscope/comparators/binary.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/diffoscope/comparators/binary.py b/diffoscope/comparators/binary.py index 9663214..2b61538 100644 --- a/diffoscope/comparators/binary.py +++ b/diffoscope/comparators/binary.py @@ -182,8 +182,8 @@ class File(object, metaclass=ABCMeta): def has_same_content_as(self, other): logger.debug('%s has_same_content %s', self, other) # try comparing small files directly first -my_size = os.path.getsize(self.path) -other_size = os.path.getsize(other.path) +my_size = os.lstat(self.path).st_size +other_size = os.lstat(other.path).st_size if my_size == other_size and my_size <= SMALL_FILE_THRESHOLD: if open(self.path, 'rb').read() == open(other.path, 'rb').read(): return True -- 2.1.4 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#818856: diffoscope: crashes on broken symlinks
Hi Reiner! Thank you for reviewing patch. I have made all the changes you mentioned above. Please find an attachment :-) Cheers, Satyam Zode From 2e440a527e4689a6f4321695801b5cd04ff8642a Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Thu, 7 Apr 2016 00:31:42 +0530 Subject: [PATCH 1/2] Fixed issue related to diffoscope symlinks crashing --- diffoscope/comparators/binary.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/diffoscope/comparators/binary.py b/diffoscope/comparators/binary.py index 9663214..2b61538 100644 --- a/diffoscope/comparators/binary.py +++ b/diffoscope/comparators/binary.py @@ -182,8 +182,8 @@ class File(object, metaclass=ABCMeta): def has_same_content_as(self, other): logger.debug('%s has_same_content %s', self, other) # try comparing small files directly first -my_size = os.path.getsize(self.path) -other_size = os.path.getsize(other.path) +my_size = os.lstat(self.path).st_size +other_size = os.lstat(other.path).st_size if my_size == other_size and my_size <= SMALL_FILE_THRESHOLD: if open(self.path, 'rb').read() == open(other.path, 'rb').read(): return True -- 2.1.4 ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
Re: [Reproducible-builds] Bug#818856: diffoscope: crashes on broken symlinks
Package: diffoscope Version: 51 Severity: normal Hi, Pabs! Please try an attached patch. Hopefully, it'll solve this problem. :) Thanks! Satyam Zode PICT, Pune On Sun, Apr 3, 2016 at 2:02 AM, Satyam Zode wrote: > Hi, Pabs! > > Please try an attached patch! Hopefully, it'll solve this problem. :) > > Thank you! > Satyam > Satyam Zode > PICT, Pune > > > On Mon, Mar 21, 2016 at 8:02 AM, Paul Wise wrote: >> Package: diffoscope >> Version: 51 >> Severity: normal >> >> I found a way to crash diffoscope with broken symlinks: >> >> pabs@chianamo ~ $ mkdir foo bar >> pabs@chianamo ~ $ touch foo/baz >> pabs@chianamo ~ $ ln -s asdasfahguwbagpiuasbdifgsabdf bar/baz >> pabs@chianamo ~ $ diffoscope foo bar >> Traceback (most recent call last): >> File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 177, in >> main >> sys.exit(run_diffoscope(parsed_args)) >> File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 148, in >> run_diffoscope >> parsed_args.file1, parsed_args.file2) >> File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", >> line 92, in compare_root_paths >> return compare_directories(path1, path2) >> File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", >> line 104, in compare_directories >> return FilesystemDirectory(path1).compare(FilesystemDirectory(path2)) >> File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", >> line 157, in compare >> my_file, other_file, source=name) >> File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", >> line 102, in compare_files >> if file1.has_same_content_as(file2): >> File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 143, in >> tool_check >> return original_function(*args, **kwargs) >> File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", >> line 186, in has_same_content_as >> other_size = os.path.getsize(other.path) >> File "/usr/lib/python3.5/genericpath.py", line 50, in getsize >> return os.stat(filename).st_size >> FileNotFoundError: [Errno 2] No such file or directory: 'bar/baz' >> >> -- System Information: >> Debian Release: stretch/sid >> APT prefers testing >> APT policy: (900, 'testing'), (860, 'testing-proposed-updates'), (850, >> 'buildd-testing-proposed-updates'), (800, 'unstable'), (790, >> 'buildd-unstable'), (700, 'experimental'), (690, 'buildd-experimental'), >> (500, 'unstable-debug'), (1, 'experimental-debug') >> Architecture: amd64 (x86_64) >> >> Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores) >> Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8) >> Shell: /bin/sh linked to /bin/dash >> Init: systemd (via /run/systemd/system) >> >> Versions of packages diffoscope depends on: >> ii python3-libarchive-c 2.1-3 >> ii python3-magic 1:5.25-2 >> ii python3-pkg-resources 18.8-1 >> pn python3:any >> >> Versions of packages diffoscope recommends: >> ii acl 2.2.52-3 >> ii binutils-multiarch 2.26-5 >> ii bzip2 1.0.6-8 >> ii caca-utils 0.99.beta19-2+b1 >> ii colord 1.2.12-1 >> ii cpio2.11+dfsg-5 >> pn default-jdk | java-sdk >> ii enjarify20151118-1 >> ii fontforge-extras0.3-4 >> pn fp-utils >> ii genisoimage 9:1.1.11-3 >> ii gettext 0.19.7-2 >> ii ghc 7.10.3-7 >> ii ghostscript 9.18~dfsg-4 >> ii gnupg 1.4.20-4 >> pn mono-utils >> ii pdftk 2.02-3 >> ii poppler-utils 0.38.0-2 >> ii python3-debian 0.1.27 >> pn python3-guestfs >> ii python3-rpm 4.12.0.1+dfsg1-3+b2 >> ii python3-tlsh3.4.4+20151206-1+b1 >> ii rpm2cpio4.12.0.1+dfsg1-3+b2 >> ii sng 1.1.0-1 >> ii sqlite3 3.11.1-1 >> ii squashfs-tools 1:4.3-3 >> ii unzip 6.0-20 >> ii vim-common 2:7.4.963-1+b2 >> ii xz-utils5.1.1alpha+20120614-2.1 >> >> Versions of packages diffos
Re: [Reproducible-builds] Bug#818856: diffoscope: crashes on broken symlinks
Hi, Pabs! Please try an attached patch! Hopefully, it'll solve this problem. :) Thank you! Satyam Satyam Zode PICT, Pune On Mon, Mar 21, 2016 at 8:02 AM, Paul Wise wrote: > Package: diffoscope > Version: 51 > Severity: normal > > I found a way to crash diffoscope with broken symlinks: > > pabs@chianamo ~ $ mkdir foo bar > pabs@chianamo ~ $ touch foo/baz > pabs@chianamo ~ $ ln -s asdasfahguwbagpiuasbdifgsabdf bar/baz > pabs@chianamo ~ $ diffoscope foo bar > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 177, in > main > sys.exit(run_diffoscope(parsed_args)) > File "/usr/lib/python3/dist-packages/diffoscope/__main__.py", line 148, in > run_diffoscope > parsed_args.file1, parsed_args.file2) > File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", > line 92, in compare_root_paths > return compare_directories(path1, path2) > File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", > line 104, in compare_directories > return FilesystemDirectory(path1).compare(FilesystemDirectory(path2)) > File "/usr/lib/python3/dist-packages/diffoscope/comparators/directory.py", > line 157, in compare > my_file, other_file, source=name) > File "/usr/lib/python3/dist-packages/diffoscope/comparators/__init__.py", > line 102, in compare_files > if file1.has_same_content_as(file2): > File "/usr/lib/python3/dist-packages/diffoscope/__init__.py", line 143, in > tool_check > return original_function(*args, **kwargs) > File "/usr/lib/python3/dist-packages/diffoscope/comparators/binary.py", > line 186, in has_same_content_as > other_size = os.path.getsize(other.path) > File "/usr/lib/python3.5/genericpath.py", line 50, in getsize > return os.stat(filename).st_size > FileNotFoundError: [Errno 2] No such file or directory: 'bar/baz' > > -- System Information: > Debian Release: stretch/sid > APT prefers testing > APT policy: (900, 'testing'), (860, 'testing-proposed-updates'), (850, > 'buildd-testing-proposed-updates'), (800, 'unstable'), (790, > 'buildd-unstable'), (700, 'experimental'), (690, 'buildd-experimental'), > (500, 'unstable-debug'), (1, 'experimental-debug') > Architecture: amd64 (x86_64) > > Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores) > Locale: LANG=en_AU.utf8, LC_CTYPE=en_AU.utf8 (charmap=UTF-8) > Shell: /bin/sh linked to /bin/dash > Init: systemd (via /run/systemd/system) > > Versions of packages diffoscope depends on: > ii python3-libarchive-c 2.1-3 > ii python3-magic 1:5.25-2 > ii python3-pkg-resources 18.8-1 > pn python3:any > > Versions of packages diffoscope recommends: > ii acl 2.2.52-3 > ii binutils-multiarch 2.26-5 > ii bzip2 1.0.6-8 > ii caca-utils 0.99.beta19-2+b1 > ii colord 1.2.12-1 > ii cpio2.11+dfsg-5 > pn default-jdk | java-sdk > ii enjarify20151118-1 > ii fontforge-extras0.3-4 > pn fp-utils > ii genisoimage 9:1.1.11-3 > ii gettext 0.19.7-2 > ii ghc 7.10.3-7 > ii ghostscript 9.18~dfsg-4 > ii gnupg 1.4.20-4 > pn mono-utils > ii pdftk 2.02-3 > ii poppler-utils 0.38.0-2 > ii python3-debian 0.1.27 > pn python3-guestfs > ii python3-rpm 4.12.0.1+dfsg1-3+b2 > ii python3-tlsh3.4.4+20151206-1+b1 > ii rpm2cpio4.12.0.1+dfsg1-3+b2 > ii sng 1.1.0-1 > ii sqlite3 3.11.1-1 > ii squashfs-tools 1:4.3-3 > ii unzip 6.0-20 > ii vim-common 2:7.4.963-1+b2 > ii xz-utils5.1.1alpha+20120614-2.1 > > Versions of packages diffoscope suggests: > ii libjs-jquery 1.11.3+dfsg-4 > > -- no debconf information > > -- > bye, > pabs > > https://wiki.debian.org/PaulWise > > > > ___ > Reproducible-builds mailing list > Reproducible-builds@lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds From 3e9aea18767099dffe62c14e7215aed54347a10f Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Mon, 21 Mar 2016 23:12:55 +0530 Subject: [PATCH 1/2] fixed issue related to diffoscope symlinks crashing --- diffoscope/comparators/bina
Re: [Reproducible-builds] [GSoC 2016] : Application review
Hi all! Jérémy Bobbio: > Satyam Zode: >> So this summer I intend to work on >> 1) Improvements to diffoscope: >> 1.1) Allow users to ignore arbitrary differences (Addition of >> ignore-profiles flag). >> 1.2) Perform fuzzy-matching across archives. >> 1.3) Finish parallel processing part. >> Above points are mentioned on GSoC wiki. And also there are more >> features mentioned in whishlist >> (https://reproducible-builds.org/events/athens2015/diffoscope-wishlist/) >> I will try to cover some of those too. > > Sounds good. Be aware that the first part will require some design work. > Finding the right UI regarding ignores might require input from the > community. I'd recommend to split this into two parts > (design+experiment+survey / implementation+documentation). You probably > will want to work on other things in parellel with the discussions. > I have started thinking on design will soon let you all know. It'd be great if you show me some directions towards designing the proposed features. (I will create new thread for this). > Could you try to come up with rough estimations on how much time all of > the above would require? > Yes! But first, I need some time for that (maybe upcoming 24 hours) because I want to think more about features proposed above(I don't know how much complex those features could be!). Moving to the interesting part :) > Fixing #818856 should not be too hard. If you could submit a patch that > would make more confident that you could do all the above. > Thank you so much for giving me this task :-) . Now I know how diffoscope can be tested and build from code. (Before this I was only familiar with codebase and diffoscope functionalities ) I have fixed this issue. I also have fixed link in the documentation. Please find an attachment. Here is the output which I have got: satyam@satyamz:~/Debian/experiment/diffoscope/bin$ mkdir foo bar satyam@satyamz:~/Debian/experiment/diffoscope/bin$ touch foo/baz satyam@satyamz:~/Debian/experiment/diffoscope/bin$ ln -s somefile bar/baz satyam@satyamz:~/Debian/experiment/diffoscope/bin$ ./diffoscope foo bar --- foo +++ bar ├── stat {} │ @@ -1,8 +1,8 @@ │ │Size: 4096 Blocks: 8 IO Block: 4096 directory │ Links: 2 │ Access: (0755/drwxr-xr-x) Uid: ( 1000/ satyam) Gid: ( 1000/ satyam) │ │ -Modify: 2016-03-21 17:22:54.004395232 + │ +Modify: 2016-03-21 17:23:20.516394537 + │ │ Birth: - │ --- foo/baz ├── +++ bar/baz │ @@ -0,0 +1,2 @@ │ +000: 6465 7374 696e 6174 696f 6e3a 2073 6f6d destination: som │ +010: 6566 696c 650a efile. │ ├── stat {} │ │ @@ -1,8 +1,8 @@ │ │ │ │ - Size: 0 Blocks: 0 IO Block: 4096 regular empty file │ │ + Size: 8 Blocks: 0 IO Block: 4096 symbolic link │ │ Links: 1 │ │ -Access: (0644/-rw-r--r--) Uid: ( 1000/ satyam) Gid: ( 1000/ satyam) │ │ +Access: (0777/lrwxrwxrwx) Uid: ( 1000/ satyam) Gid: ( 1000/ satyam) │ │ │ │ -Modify: 2016-03-21 17:22:54.004395232 + │ │ +Modify: 2016-03-21 17:23:20.516394537 + │ │ │ │ Birth: - │ ╵ ╵ satyam@satyamz:~/Debian/experiment/diffoscope/bin$ > Most of the improvements we could think of have indeed been implemented > since. :) Nice :) I also want to know, whether I will be able to edit application or not after a deadline? Thanks again! Satyam Zode From 3e9aea18767099dffe62c14e7215aed54347a10f Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Mon, 21 Mar 2016 23:12:55 +0530 Subject: [PATCH 1/2] fixed issue related to diffoscope symlinks crashing --- diffoscope/comparators/binary.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/diffoscope/comparators/binary.py b/diffoscope/comparators/binary.py index 9663214..5622a9c 100644 --- a/diffoscope/comparators/binary.py +++ b/diffoscope/comparators/binary.py @@ -183,7 +183,7 @@ class File(object, metaclass=ABCMeta): logger.debug('%s has_same_content %s', self, other) # try comparing small files directly first my_size = os.path.getsize(self.path) -other_size = os.path.getsize(other.path) +other_size = os.lstat(other.path).st_size if my_size == other_size and my_size <= SMALL_FILE_THRESHOLD: if open(self.path, 'rb').read() == open(other.path, 'rb').read(): return True -- 2.1.4 From 79809c35a402f1e28f1c3f7c94985274172c0415 Mon Sep 17 00:00:00 2001 From: Satyam Zode Date: Mon, 21 Mar 2016 23:25:27 +0530 Subject: [PATCH 2/2] Fixed link in documentation --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index fe23a37..5288e52 100644 --- a/README.rst +++ b/README.rst @@ -83,7 +83,7 @@ system against the diffoscope package: Join the users and developers mailin
Re: [Reproducible-builds] [GSoC 2016] : Application review
Hi, everyone! Jérémy Bobbio: > Thanks for your application. I much appreciated that it's done before > the deadline. I also like you being clear of your other commitments. > > I think it would have been fine application for last summer, but we've > made significant progress on several fronts since, therefore I'm not > convinced that there's much work left in the tasks your propose. I think > it would be better to aim with more precise tasks, e.g. toolchain > software you'll be improving to solve classes of issues, or at leas an > outline the features you intend to add to strip-nondeterminism or > diffoscope. > > (If you feel you're part of the reproducible builds team and disagree > with my comments, please say so!) Thanks Lunar for this valuable feedback. Yes, I am agree with you. After reading the reasons( which you mentioned) and as a part of reproducible builds team I don't think the proposed work(don't need whole summer to work on) by me will help much to reproducible builds effort too. But I think there are some issues which still needs to be fixed. There are some issues in which not even a single package have patch. I will try to look into those and will try to search for solutions. So this summer I intend to work on 1) Improvements to diffoscope: 1.1) Allow users to ignore arbitrary differences (Addition of ignore-profiles flag). 1.2) Perform fuzzy-matching across archives. 1.3) Finish parallel processing part. Above points are mentioned on GSoC wiki. And also there are more features mentioned in whishlist (https://reproducible-builds.org/events/athens2015/diffoscope-wishlist/) I will try to cover some of those too. I guess Better/smarter ELF diffing is underdevelopment (I have checked git logs and diffoscope for same) 2) Improving reproducibility of Debian packages: In this section I will be fixing Debian packages and will try to find the solutions to the issues which do not have solution yet. I am trying to enlist such issues. > If you look at packages identified as leaving timestamps in gzip > headers, you'll see that most of them already have patches, and the ones > who don't are affected with other issues > https://tests.reproducible-builds.org/issues/unstable/timestamps_in_gzip_headers_issue.html > These other issues probably deter maintainers' motivation to fix the > problems with gzip timestamps. > > Almost all packages with varying mtimes in data.tar or control.tar have > patches or have been fixed through toolchain improvements: > https://tests.reproducible-builds.org/issues/unstable/varying_mtimes_in_data_tar_gz_or_control_tar_gz_issue.html > > It feels quite suboptimal to highlight user and groups in tarballs as > separate issues as I think all are affected by other tarball related > issues. They should be fixed at the same time: > https://tests.reproducible-builds.org/issues/unstable/users_and_groups_in_tarball_issue.html > > Regarding timestamps due to C pre-processor macros, Dhole is waiting > for GCC patch window to open again—which will be in April, IIRC. > So unless you intend to work on adding support for SOURCE_DATE_EPOCH in > clang, I'm not sure there's much work left on this issue. I believe that > fixing the 400+ packages individually should not be undertaken if > we can avoid it. > https://gcc.gnu.org/ml/gcc-patches/2015-12/msg01402.html > https://wiki.debian.org/ReproducibleBuilds/TimestampsFromCPPMacros > > Emmanuel Bourg has been working and fixing almost all Java-related > issues in the course of the past year. I expect he'll probably work on > this fixing locale related javadoc issue in a near future. I guess you > could coordinate with him to write the necessary patches, though. > https://tests.reproducible-builds.org/issues/unstable/locale_in_documentation_generated_by_javadoc_issue.html > A big thanks to you, because I really didn't know about many of the above things. Its good to know that people are already working on this part. :-) > These quick evaluations leave me the feeling that your proposed schedule > is currently not adequate with actual needs of the reproducible builds > effort. > > This probably means that progress can be made on making more visible > areas that actually require work… Please let me know what you think about the work which I have proposed now. I will frame timeline accordingly. PS: I really feel that I am part of reproducible builds and I want to strengthen the bond by spending my summer working with reproducible builds ;-) ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds
[Reproducible-builds] Bug#818111: debian-keyring: please make the build reproducible (locale, fileordering)
Source: debian-keyring Version: 2016.01.20 Severity: wishlist Tags: patch User: reproducible-builds@lists.alioth.debian.org Usertags: fileordering X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org Hi! While working on the “reproducible builds” effort [1], we have noticed that debian-keyring could not be built reproducibly. The attached patch fix the order of files in md5sums. Once applied, debian-keyring can be built reproducibly in our current experimental framework. [1]: https://wiki.debian.org/ReproducibleBuilds Thank you! Satyam Zode PICT, Pune diff -Nru debian-keyring-2015.04.10/debian/changelog debian-keyring-2015.04.10.0~reproducible1/debian/changelog --- debian-keyring-2015.04.10/debian/changelog 2015-04-10 14:53:32.0 +0530 +++ debian-keyring-2015.04.10.0~reproducible1/debian/changelog 2016-03-13 23:59:02.0 +0530 @@ -1,3 +1,9 @@ +debian-keyring (2015.04.10.0~reproducible1) UNRELEASED; urgency=medium + + * Made debian-keyring reproducible + + -- Satyam Zode Sun, 13 Mar 2016 23:57:03 +0530 + debian-keyring (2015.04.10) unstable; urgency=medium [ Gunnar Wolf ] diff -Nru debian-keyring-2015.04.10/debian/rules debian-keyring-2015.04.10.0~reproducible1/debian/rules --- debian-keyring-2015.04.10/debian/rules 2015-03-31 04:23:17.0 +0530 +++ debian-keyring-2015.04.10.0~reproducible1/debian/rules 2016-03-14 00:01:43.0 +0530 @@ -53,7 +53,8 @@ chown -R root.root debian/tmp chmod -R go=rX debian/tmp - cd debian/tmp && find . -type f ! -regex '.*DEBIAN/.*' -printf '%P\0' | xargs -r0 md5sum > DEBIAN/md5sums + cd debian/tmp && find . -type f ! -regex '.*DEBIAN/.*' -printf '%P\0' | \ + LC_ALL=C sort -z | xargs -r0 md5sum > DEBIAN/md5sums find debian/tmp -depth -newermt '$(BUILD_DATE)' -print0 | \ xargs -0r touch --no-dereference --date='$(BUILD_DATE)' ___ Reproducible-builds mailing list Reproducible-builds@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds