Re: mock, megadeps and PyPI
- Original Message - From: Nick Coghlan ncogh...@redhat.com To: Fedora Python SIG python-devel@lists.fedoraproject.org Sent: Thursday, August 21, 2014 2:35:48 AM Subject: mock, megadeps and PyPI Jeff Fearn reminded me today of mock --megadeps, a patched version of mock he created that supports recursively downloading and building dependencies in a chroot, without incurring the overhead of setting up and tearing down multiple mock build environments the way the chainbuild command does. The mock RFE is at https://bugzilla.redhat.com/show_bug.cgi?id=843745, while Jeff is now maintaining the patched version at https://github.com/jfearn/mock I'm bringing this to folks attention mainly due to one of the neat features it has embedded in it, which is integration with CPAN: it can use cpanspec to create an SRPM on the fly. That's likely not going to be of suitable quality for Fedora itself, but it should be good enough for COPR and potentially even Playground. My question is whether it would be practical to do something similar with pyp2rpm. Bonus points if we can come up with a way to integrate it nicely with COPR, or even tidy up the implementation to the point where we can convince Clark to accept the feature as part of mock itself :) Hi Nick, I have created vague named issue[0] at pyp2rpm repo we can discuss it there. I will post my opinion later, next week probably, currently I am off from work. Regards, Nick. -- Nick Coghlan Red Hat Hosted Shared Services Software Engineering Development, Brisbane HSS Provisioning Architect ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel Regards, Robert Kuska - rkuska @ #fedora-devel on freenode #brno #gulag #software-collections on brq.redhat [0] https://bitbucket.org/bkabrda/pyp2rpm/issue/13 ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Re: reproducible builds and python
- Original Message -
Hi everyone!
I've been doing some work towards reproducible builds in Fedora (mostly
with various upstreams so far) and one of the elephants in the Room are
obviously Pythons .pyc and .pyo files.
As those contain the mtime of the original .py file, they might be
different for each rebuild of an srpm.
For many rpms this isn't a problem, because the files are not modified
and thus retain their timestamp from the archive. Quite a few rpms do
modify to .py files though and because of that, every build has a
different result.
I would like to propose to set the mtime of all .py files to a fixed
(for this specific srpm) time. This could be done
in /usr/lib/rpm/brp-python-bytecompile before doing the actual
byte-compilation. This would result in the same .py{c,o} files being
created for each rebuild.
The timestamp could be e.g. the mtime of the oldest file in the
buildroot (which would assume that not _all_ of the files are modified)
But if you are interested in the idea, I'd certainly be open to
suggestions.
Generally, I like this idea, but I have some concerns:
- So the bytecompile script would touch all *.py files? It seems a bit hacky,
not mentioning that in some specfiles (notably python3 itself) we actually have
to do bytecompilation by hand for certain reasons.
- Obviously another question is what happens when _all_ files are modified. I
can pretty much guarantee you that at any given time there will be at least one
package in Fedora that will have all files modified (e.g. python-six has just
one py file, so if we patch/touch it in some way, the problem is here). I'd
like to see a proposal that handles this situation in a sane way.
Having {read about,experimented with} reproducible builds before, I can see the
advantage that Fedora would get from this. Perhaps you could sum up the actual
benefits of reproducible builds here so that even those who have never heard of
this can see why this is worthwile?
Just thinking aloud here, but this should also be beneficial for RPMs generated
with setup.py bdist_rpm, right? As in two RPMs generated by bdist_rpm from
the same git/hg revision on the same architecture would have the same hash -
or am I wrong here?
Thanks,
Slavek
To address the obvious question:
Why not special-case those files when comparing rpms?
It will certainly be impossible to achieve this for all packages in
Fedora, so for some files this might indeed be needed, but I think we
should avoid this where possible. The idea of reproducible builds
becomes meaningless if the amount of differences that you just ignore
gets to big.
What do you think of this proposal?
Greetings,
Benedikt
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel
--
Regards,
Slavek Kabrda
___
python-devel mailing list
python-devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/python-devel
Re: mock, megadeps and PyPI
- Original Message - - Original Message - From: Nick Coghlan ncogh...@redhat.com To: Fedora Python SIG python-devel@lists.fedoraproject.org Sent: Thursday, August 21, 2014 2:35:48 AM Subject: mock, megadeps and PyPI Jeff Fearn reminded me today of mock --megadeps, a patched version of mock he created that supports recursively downloading and building dependencies in a chroot, without incurring the overhead of setting up and tearing down multiple mock build environments the way the chainbuild command does. The mock RFE is at https://bugzilla.redhat.com/show_bug.cgi?id=843745, while Jeff is now maintaining the patched version at https://github.com/jfearn/mock I'm bringing this to folks attention mainly due to one of the neat features it has embedded in it, which is integration with CPAN: it can use cpanspec to create an SRPM on the fly. That's likely not going to be of suitable quality for Fedora itself, but it should be good enough for COPR and potentially even Playground. My question is whether it would be practical to do something similar with pyp2rpm. Bonus points if we can come up with a way to integrate it nicely with COPR, or even tidy up the implementation to the point where we can convince Clark to accept the feature as part of mock itself :) Hi Nick, I have created vague named issue[0] at pyp2rpm repo we can discuss it there. I will post my opinion later, next week probably, currently I am off from work. I like this idea. Not only it can be beneficial for people creating a whole environment for their Python application, but we would also be able to get a huge feedback on pyp2rpm and would be able to improve it significantly, I'd say. /me is wondering whether we could automatically generate SCLs by this, so that the packages wouldn't conflict with system ones. In the best case, one could even be able to do mock-or-a-mock-wrapper --create-scl-from-this-app-and-build-it-in-copr /path/to/my/app scl_name (I admit that naming command line arguments isn't my strongest skill...) Slavek Regards, Nick. -- Nick Coghlan Red Hat Hosted Shared Services Software Engineering Development, Brisbane HSS Provisioning Architect ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel Regards, Robert Kuska - rkuska @ #fedora-devel on freenode #brno #gulag #software-collections on brq.redhat [0] https://bitbucket.org/bkabrda/pyp2rpm/issue/13 ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel -- Regards, Slavek Kabrda ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
Re: mock, megadeps and PyPI
Dne 21.8.2014 09:40, Bohuslav Kabrda napsal(a): I like this idea. Not only it can be beneficial for people creating a whole environment for their Python application, but we would also be able to get a huge feedback on pyp2rpm and would be able to improve it significantly, I'd say. /me is wondering whether we could automatically generate SCLs by this, so that the packages wouldn't conflict with system ones. In the best case, one could even be able to do mock-or-a-mock-wrapper --create-scl-from-this-app-and-build-it-in-copr /path/to/my/app scl_name (I admit that naming command line arguments isn't my strongest skill...) Might also be usable for packagers trying to build something against the Python 3.5 SCL [1]. [1] http://copr.fedoraproject.org/coprs/churchyard/python3-nightly/ -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ python-devel mailing list python-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/python-devel
