[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: I think this issue can be closed, since Martin's touch step runs on the bots successfully, and the ASDL dependencies in .hgtouch were fixed. -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Roundup Robot added the comment: New changeset 56ed149e597a by Eli Bendersky in branch 'default': Mention 'make touch' in the devguide. http://hg.python.org/devguide/rev/56ed149e597a -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: I've updated the devguide about `make touch`. As for the buildbot configuration, I agree we shouldn't document a static snapshot and also there's the passwords problem. I was talking to Antoine about this the other day, and maybe there's sense to create an infrastructure section in the devguide that will explain how to log into the buildbot server and look at the configuration. This should be enough for folks who really need to figure it out and is also safe w.r.t. getting out of date and exposing passwords. Also, IMHO #15964 can be closed. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Antoine Pitrou added the comment: I was talking to Antoine about this the other day, and maybe there's sense to create an infrastructure section in the devguide that will explain how to log into the buildbot server and look at the configuration. IMHO it would make more sense to create a separate infrastructure guide. There's no need to bloat the devguide with such information, which could quickly become quite complicated. -- title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots - Prepare .hgtouch and Tools/hg/hgtouch.py to run on thebots ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: I'm not sure a separate document is right here because that's one more repository to have. The devguide already contains sections for somewhat more esoteric things like compiler internals guide, coverity scans. Besides, section 18 already has some material about the buildbots. But I don't feel strongly enough about this to argue :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Antoine Pitrou added the comment: I'm not sure a separate document is right here because that's one more repository to have. The devguide already contains sections for somewhat more esoteric things like compiler internals guide, coverity scans. Besides, section 18 already has some material about the buildbots. Well, ideally the more esoteric things wouldn't really be there. Perhaps other people have different opinions? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: On Tue, Oct 1, 2013 at 7:00 AM, Antoine Pitrou rep...@bugs.python.orgwrote: Antoine Pitrou added the comment: I'm not sure a separate document is right here because that's one more repository to have. The devguide already contains sections for somewhat more esoteric things like compiler internals guide, coverity scans. Besides, section 18 already has some material about the buildbots. Well, ideally the more esoteric things wouldn't really be there. I still see the current situation better than the previous one, where this was scattered around PEPs, Wikis and just random web pages. Having it all in the same repository is not ideal from a taxonomical point of view, perhaps, but as a single collection of everything related to the development *of* Python it's not too bad since keeping it in a single place makes it easier to maintain and review; mostly, there's a somewhat smaller chance that things will be forgotten and rot. To put it differently, maintaining the devguide is kind-of a chore for devs as it is; splitting it into multiple topical repositories will just make the burden higher so we'll end up not maintaining it at all :) Perhaps other people have different opinions? -- title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots - Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: +nick,guido,benjamin: in case you're interested in the discussion that takes place in the most recent messages of this issue -- nosy: +benjamin.peterson, gvanrossum, ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Martin v. Löwis added the comment: Documenting the pydotorg setup is a field of ongoing discussions. If you ask three people involved, you get three answers where this should be documented, with no chance for consensus. From an infrastructure point of view, it is up to the infrastructure head (i.e. Noah Kantrowitz) to rule on where this should be documented. I can see the python-dev view, to, although the documentation might be as short as ask Noah. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Nick Coghlan added the comment: Where to document the infrastructure setup (including the buildbots) is up to Noah/infrastructure-sig, as the devguide just covers CPython development, while the PSF infrastructure team covers a lot more than that. A pointer from the devguide docs to the appropriate infrastructure docs would make sense, but it isn't the right home for the docs themselves. The interim solution I use is ask on python-dev or ask on python-committers :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Antoine Pitrou added the comment: I still see the current situation better than the previous one, where this was scattered around PEPs, Wikis and just random web pages. Having it all in the same repository is not ideal from a taxonomical point of view, perhaps, but as a single collection of everything related to the development *of* Python it's not too bad since keeping it in a single place makes it easier to maintain and review; Well, precisely, it's putting everything in a single collection that I find bad, because it could become dreadful for beginner contributors. The devguide is primarily meant to ensure contributors don't get lost, even though it can serve as a reminder for core developers too. -- title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots - Prepare .hgtouch and Tools/hg/hgtouch.py to run on thebots ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Martin v. Löwis added the comment: I see two solutions, both involving to track the newest timestamp that needs to pass. a) touch the file to have a time stamp in the future. make might complain, though. b) sleep until a second has passed, then touch (actually, sleep of 100ms multiple times until the time stamp is new enough) The key issue is that hg touch should use the same comparison as make. So if make thinks that something need to be done, hg touch should touch the files. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Martin v. Löwis added the comment: Here is a patch that backdates outputs 1s after their youngest input (and sleeps if necessary to avoid producing files in the future). With that, I get lap-le:3k loewis$ touch Parser/asdl_c.py ;date;make touch;date Mo 30 Sep 2013 15:29:21 CEST hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Touching Include/Python-ast.h Touching Python/Python-ast.c Mo 30 Sep 2013 15:29:23 CEST lap-le:3k loewis$ ls -lT Parser/asdl_c.py Include/Python-ast.h Python/Python-ast.c -rw-r--r-- 1 loewis admin 19489 30 Sep 15:29:22 2013 Include/Python-ast.h -rwxr-xr-x 1 loewis admin 43732 30 Sep 15:29:21 2013 Parser/asdl_c.py -rw-r--r-- 1 loewis admin 228785 30 Sep 15:29:23 2013 Python/Python-ast.c BTW, I think the Python-ast.c issue could be solved by just having Python-ast.c depend on the same inputs as Python-ast.h; the C file isn't actually depending on the H file. -- keywords: +patch Added file: http://bugs.python.org/file31920/touch.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: LGTM -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Roundup Robot added the comment: New changeset 477246839224 by Martin v. Löwis in branch '3.3': Issue #19106: Touch generated files to be 1s newer than their youngest source. http://hg.python.org/cpython/rev/477246839224 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Roundup Robot added the comment: New changeset 86eff5c4e698 by Martin v. Löwis in branch '3.3': Issue #19106: Add buildbottouch target. http://hg.python.org/cpython/rev/86eff5c4e698 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Martin v. Löwis added the comment: I have now updated the master.cfg to make buildbottouch a separate build step. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: With your fix, `make touch` now behaves as expected. Also, I can see the step added to the bots (e.g. http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/2597/steps/shell/logs/stdio) Thanks, I think the issue is resolved then. We should now be able to have modern Python code in scripts that generate code such as asdl[_c].py because bots and people will only old versions of Python don't ever need to run them. Should we add a note about this to the dev guide? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Martin v. Löwis added the comment: make touch (or hg touch) certainly should be mentioned. Details of the buildbot configuration need not. Publishing the buildbot configuration file might be useful, except that it also contains all the builder passwords, which should not be published. If that is done, it should be the life configuration file (rather than a static copy created now). It might be easiest to publish it through a symlink on buildbot.python.org. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
New submission from Eli Bendersky: Background: we'd like to enable running 'make touch' on the bots before 'make -jN', to avoid problems with scripts that auto-generate code for the Python build (bootstrapping). pydev thread: https://mail.python.org/pipermail/python-dev/2013-September/128992.html A couple of issues are prerequisites: 1. Fix the asdl dependencies in .hgtouch: Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py Python/Python-ast.c: Include/ast.h The file Include/ast.h is not, in fact, auto-generated. But Include/Python-ast.h *is*, and it does not appear in this file. 2. Make sure 'hg touch' itself runs on Python 2.4 because that's what some bots have (and other older systems like RHEL 5) -- assignee: eli.bendersky components: Build messages: 198482 nosy: eli.bendersky, ezio.melotti, loewis, pitrou priority: normal severity: normal status: open title: Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots type: behavior versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: .hgtouch fixed in ac19ff225280 (I specified the issue number incorrectly so this one wasn't notified). Curiously, `make touch` seems to think there's still things to do even after the first round of touching: $ touch Parser/asdl_c.py $ make touch hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Touching Include/Python-ast.h Touching Python/Python-ast.c $ make touch hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Touching Python/Python-ast.c -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: Ooh, I think that's because the # try processing all rules in topological order in do_touch doesn't actually topo-sort. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19106] Prepare .hgtouch and Tools/hg/hgtouch.py to run on the bots
Eli Bendersky added the comment: The problem in this case is different, actually. It's the comparison: if o_time = i_time: # generated file is older, touch need_touch = True In check_rule. The script is pretty quick so when it touches both Python-ast.h and .c they get the same stat time exactly. In the next run, then, this comparison succeeds and .c is touched again. I'm not sure what's the right way to go about this? Changing the comparison to may theoretically miss cases in which the input was updated an epsilon after it auto-generated its output, and the change will go unnoticed. A different solution would be to introduce a micro-wait between each 'touch' to make sure that transitive dependencies don't need to be revisited in the future. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19106 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
