In perl.git, the branch blead has been updated <http://perl5.git.perl.org/perl.git/commitdiff/abec7027d3cfd4d003895131f8b4ec25ca774a18?hp=e34fc33f2642e698df23a282ac508fe9214ea68a>
- Log ----------------------------------------------------------------- commit abec7027d3cfd4d003895131f8b4ec25ca774a18 Merge: e34fc33f26 bd4de633ba Author: Aaron Crane <[email protected]> Date: Fri Dec 30 14:01:57 2016 +0000 [MERGE] Improve Porting/sync-with-cpan The original goal of this sequence of patches was to address RT#130375. That ticket proposes extending the release manager's guide with discussion of what regen/lib_cleanup.pl does, and why it might sometimes be needed, and thus why Porting/sync-with-cpan might yield a porting-test failure when t/porting/regen.t detects that it needs to be run. However, I find the RMG is already relatively bulky, and therefore hard to navigate; so adding more text to it would exacerbate that problem. In addition, it seems much better to merely fix this problem: we can always run regen/lib_cleanup.pl, and if it makes any changes, those changes can be committed along with the updated CPAN module being imported. This merge also contains a few other improvements to Porting/sync-with-cpan: - The log file it generates is now listed in .gitignore - It allows specifying the module name with "-" rather than "::" as a name separator - A bug that prevented the new module's tests from being run is now fixed - The MANIFEST is now rewritten to accommodate any file deletions introduced by the updated CPAN module - The messages emitted have been reworked slightly for clarity; in particular, "Fixing MANIFEST" will no longer be followed by a confusing and scary "'MANIFEST' is NOT sorted properly" message - It now takes a --jobs option, which if supplied is used both for executing make and running the module's tests - The --tarball option now works when the tarball isn't in the current directory - It runs a few additional preflight checks that detect mistakes I found myself making These changes have been tested by using Porting/sync-with-cpan to import Test::Simple into blead as of 54f6f377a29af7d60918c003fc2c462439c340bd, since that was the triggering case for RT#130375. commit bd4de633baffb94fd120c5c399e69a6ac999156b Author: Aaron Crane <[email protected]> Date: Fri Dec 30 13:51:04 2016 +0000 Porting/sync-with-cpan: allow out-of-tree --tarball file Previously, in this situation, we correctly used Archive::Tar to extract the contents of the tarball under cpan/, but then tried to look for the files in a nonexistent directory alongside the original tarball. M Porting/sync-with-cpan commit 861c67962b02d8e31b9ab10311c73104f547e517 Author: Aaron Crane <[email protected]> Date: Fri Dec 30 13:46:11 2016 +0000 Porting/sync-with-cpan: more preflight checks for --tarball If the file supplied doesn't exist (or names a directory), we'll get a failure from Archive::Tar after having already made some changes. M Porting/sync-with-cpan commit d11e29912382510f8a860432049f19d4ae72c228 Author: Aaron Crane <[email protected]> Date: Fri Dec 30 13:42:51 2016 +0000 Porting/sync-with-cpan: avoid unless/else M Porting/sync-with-cpan commit 4b7602461774ae6d29af82426065893631a26e18 Author: Aaron Crane <[email protected]> Date: Fri Dec 30 11:33:37 2016 +0000 Porting/sync-with-cpan: bail out early if Configure hasn't been run M Porting/sync-with-cpan commit 215d9c65023489d6830558a77d96a52186d001fa Author: Aaron Crane <[email protected]> Date: Fri Dec 30 11:32:39 2016 +0000 Porting/sync-with-cpan: factor out internal WIN32 constant M Porting/sync-with-cpan commit 06998c55f9d56cbee761af0d6dc5ec06fcee3c62 Author: Aaron Crane <[email protected]> Date: Fri Dec 30 11:51:38 2016 +0000 Porting/sync-with-cpan: apply --jobs=N when running module tests This requires switching from t/TEST to t/harness. M Porting/sync-with-cpan commit b7078f1e50d0dc6bb22fc141b4d47eab53279446 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 19:01:51 2016 +0000 Porting/sync-with-cpan: add --jobs option M Porting/sync-with-cpan commit d9d83ea59de794d098efc0d7171c66af36dca4ca Author: Aaron Crane <[email protected]> Date: Thu Dec 29 18:54:56 2016 +0000 Porting/sync-with-cpan: avoid "Attempting to â¦" message That message doesn't have a follow-up along the lines of "and it worked", so the user might be left wondering what happened. M Porting/sync-with-cpan commit 592f3827ae127049c6e5f14186f59609ae7b8a8f Author: Aaron Crane <[email protected]> Date: Thu Dec 29 18:40:54 2016 +0000 Porting/sync-with-cpan: improve advice emitted at end M Porting/sync-with-cpan commit d8a823f405d6ff75c47de981c528dd98be9e3979 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 18:17:45 2016 +0000 Porting/sync-with-cpan: run regen/lib_cleanup.pl if possibly needed M Porting/sync-with-cpan commit 190c1b3b7586176def76711b55b4da7c5327fa67 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 16:51:09 2016 +0000 Porting/sync-with-cpan: delete entries from MANIFEST if needed M Porting/sync-with-cpan commit c0767453776457afc3befd70130221694d2371a9 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 16:37:52 2016 +0000 Porting/sync-with-cpan: run manisort with --quiet Without this, the user is likely to see consecutive lines saying "Fixing MANIFEST" and "'MANIFEST' is NOT sorted properly", which looks pretty scary. M Porting/sync-with-cpan commit c0e3c9decf636c85524d06da42094b8c61da7053 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 16:35:10 2016 +0000 Porting/sync-with-cpan: run manisort under the current perl M Porting/sync-with-cpan commit 57b5d6e120f1ac6da4b0ec49224c35d18eb4330e Author: Aaron Crane <[email protected]> Date: Thu Dec 29 13:07:35 2016 +0000 Porting/sync-with-cpan: do in fact run the new dist's tests At the point we look for the dist's test files, we've just chdir-ed into Perl's t/ directory (since we need to be able to run the ./perl binary in that directory), so the tests are therefore under ../cpan/$dist. M Porting/sync-with-cpan commit 24c7e242df9ccb3c2da28b617dc6549607a90f82 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 13:04:36 2016 +0000 Porting/sync-with-cpan: accept dist name instead of package name Since the directories in cpan/ are named for the dist, this is perhaps more obvious, at least in some situations. M Porting/sync-with-cpan commit e6e4cae9e2d8a86847390ab67822c11db16aa126 Author: Aaron Crane <[email protected]> Date: Thu Dec 29 13:01:28 2016 +0000 Porting/sync-with-cpan: mention generation of make.log And factor that filename out into a variable. M Porting/sync-with-cpan commit af8c53c36777b42890203af0f8e0808a25bb2782 Author: Aaron Crane <[email protected]> Date: Fri Dec 30 10:18:05 2016 +0000 Porting/sync-with-cpan: don't reimplement tmpdir() M Porting/sync-with-cpan commit ba4f4bf097fc0f7337c75165e7e5cab0c82cea9d Author: Aaron Crane <[email protected]> Date: Thu Dec 29 12:57:17 2016 +0000 gitignore build product from Porting/sync-with-cpan M .gitignore ----------------------------------------------------------------------- Summary of changes: .gitignore | 3 ++ Porting/sync-with-cpan | 144 ++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 116 insertions(+), 31 deletions(-) diff --git a/.gitignore b/.gitignore index 157390afdb..131b1e4716 100644 --- a/.gitignore +++ b/.gitignore @@ -199,3 +199,6 @@ GPATH GRPATH GRTAGS GTAGS + +# generated by Porting/sync-with-cpan +/make.log diff --git a/Porting/sync-with-cpan b/Porting/sync-with-cpan index 6484c179c9..e45b1f3bc8 100755 --- a/Porting/sync-with-cpan +++ b/Porting/sync-with-cpan @@ -66,7 +66,7 @@ Restore files mentioned in C<CUSTOMIZED> =item * -Adds new files to F<MANIFEST> +Updates the contents of F<MANIFEST> =item * @@ -90,13 +90,19 @@ Runs the porting tests C<--tarball> should be the path to the tarball; the version is extracted from the filename -- but can be overwritten by the C<--version> option. -=head1 TODO +=head1 OPTIONS =over 4 -=item * +=item C<--jobs> I<N> + +When running C<make>, pass a C<< -jI<N> >> option to it. + +=back + +=head1 TODO -Delete files from F<MANIFEST> +=over 4 =item * @@ -128,15 +134,25 @@ use strict; use warnings; use Getopt::Long; use Archive::Tar; +use File::Basename qw( basename ); use File::Path qw( remove_tree ); use File::Find; +use File::Spec::Functions qw( tmpdir ); use Config qw( %Config ); $| = 1; +use constant WIN32 => $^O eq 'MSWin32'; + die "This does not look like a top level directory" unless -d "cpan" && -d "Porting"; +# Check that there's a Makefile, if needed; otherwise, we'll do most of our +# work only to fail when we try to run make, and the user will have to +# either unpick everything we've done, or do the rest manually. +die "Please run Configure before using $0\n" + if !WIN32 && !-f "Makefile"; + our @IGNORABLE; our %Modules; @@ -144,9 +160,11 @@ use autodie; require "Porting/Maintainers.pl"; +my $MAKE_LOG = 'make.log'; + my %IGNORABLE = map {$_ => 1} @IGNORABLE; -my $tmpdir= $ENV{ TEMP } // '/tmp'; +my $tmpdir = tmpdir(); my $package = "02packages.details.txt"; my $package_url = "http://www.cpan.org/modules/$package";; @@ -166,6 +184,7 @@ sub usage GetOptions ('tarball=s' => \my $tarball, 'version=s' => \my $version, + 'jobs=i' => \my $make_jobs, force => \my $force, help => sub { usage 0; }, ) or die "Failed to parse arguments"; @@ -194,20 +213,33 @@ sub de_exec { sub make { my @args= @_; - if( $^O eq 'MSWin32') { + unshift @args, "-j$make_jobs" if defined $make_jobs; + if (WIN32) { chdir "Win32"; - system "$Config{make} @args> ..\\make.log 2>&1" and die "Running make failed, see make.log"; + system "$Config{make} @args> ..\\$MAKE_LOG 2>&1" + and die "Running make failed, see $MAKE_LOG"; chdir '..'; } else { - system "$Config{make} @args> make.log 2>&1" and die "Running make failed, see make.log"; + system "$Config{make} @args> $MAKE_LOG 2>&1" + and die "Running make failed, see $MAKE_LOG"; }; }; my ($module) = shift; -my $cpan_mod = @ARGV ? shift : $module; +my $info = $Modules{$module}; +if (!$info) { + # Maybe the user said "Test-Simple" instead of "Test::Simple". See if we + # can fix it up. + (my $guess = $module) =~ s/-/::/g; + $info = $Modules{$guess} + or die "Cannot find module $module"; + say "Guessing you meant $guess instead of $module"; + $module = $guess; +} + +my $cpan_mod = @ARGV ? shift : $module; -my $info = $Modules {$module} or die "Cannot find module $module"; my $distribution = $$info {DISTRIBUTION}; my @files = glob $$info {FILES}; @@ -238,7 +270,13 @@ if ($cpan_mod =~ /-/ && $cpan_mod !~ /::/) { # my $new_file; my $new_version; -unless ($tarball) { +if (defined $tarball) { + die "Tarball $tarball does not exist\n" if !-e $tarball; + die "Tarball $tarball is not a plain file\n" if !-f _; + $new_file = $tarball; + $new_version = $version // ($new_file =~ /-([0-9._]+(?:-TRIAL[0-9]*)?)\.tar\.gz/) [0]; +} +else { # # Poor man's cache # @@ -273,10 +311,6 @@ unless ($tarball) { 1 } or system wget => $url, '-qO', $new_file; } -else { - $new_file = $tarball; - $new_version = $version // ($new_file =~ /-([0-9._]+(?:-TRIAL[0-9]*)?)\.tar\.gz/) [0]; -} my $old_dir = "$pkg_dir-$old_version"; @@ -286,7 +320,7 @@ system git => 'clean', '-dfxq', $pkg_dir; say "Unpacking $new_file"; Archive::Tar->extract_archive( $new_file ); -(my $new_dir = $new_file) =~ s/\.tar\.gz//; +(my $new_dir = basename($new_file)) =~ s/\.tar\.gz//; # ensure 'make' will update all files my $t= time; for my $file (find_type_f($new_dir)) { @@ -439,23 +473,58 @@ if ($$info {CUSTOMIZED}) { } chdir ".."; -if (@commit) { +if (@commit || @gone) { say "Fixing MANIFEST"; - my $MANIFEST = "MANIFEST"; - my $MANIFEST_SORT = "$MANIFEST.sorted"; - open my $fh, ">>", $MANIFEST; - say $fh "cpan/$pkg_dir/$_" for @commit; - close $fh; - system perl => "Porting/manisort", '--output', $MANIFEST_SORT; - rename $MANIFEST_SORT => $MANIFEST; + my $MANIFEST = "MANIFEST"; + my $MANIFEST_NEW = "$MANIFEST.new"; + + open my $orig, "<", $MANIFEST + or die "Failed to open $MANIFEST for reading: $!\n"; + open my $new, ">", $MANIFEST_NEW + or die "Failed to open $MANIFEST_NEW for writing: $!\n"; + my %gone = map +("cpan/$pkg_dir/$_" => 1), @gone; + while (my $line = <$orig>) { + my ($file) = $line =~ /^(\S+)/ + or die "Can't parse MANIFEST line: $line"; + print $new $line if !$gone{$file}; + } + + say $new "cpan/$pkg_dir/$_" for @commit; + + close $new or die "Can't close $MANIFEST: $!\n"; + + system $^X => "Porting/manisort", '--quiet', "--output=$MANIFEST", $MANIFEST_NEW; + unlink $MANIFEST_NEW + or die "Can't delete temporary $MANIFEST_NEW: $!\n"; } -print "Running a make ... "; +print "Running a make and saving its output to $MAKE_LOG ... "; # Prepare for running (selected) tests make 'test-prep'; print "done\n"; +# The build system installs code from CPAN dists into the lib/ directory, +# creating directories as needed. This means that the cleaning-related rules +# in the Makefile need to know which directories to clean up. The Makefile +# is generated by Configure from Makefile.SH, so *that* file needs the list +# of directories. regen/lib_cleanup.pl is capable of automatically updating +# the contents of Makefile.SH (and win32/Makefile, which needs similar but +# not identical lists of directories), so we can just run that (using the +# newly-built Perl, as is done with the regen programs run by "make regen"). +# +# We do this if any files at all have been added or deleted, regardless of +# whether those changes result in any directories being added or deleted, +# because the alternative would be to replicate the regen/lib_cleanup.pl +# logic here. That's fine, because regen/lib_cleanup.pl is idempotent if run +# repeatedly. +if (@commit || @gone) { + say "Running regen/lib_cleanup.pl to handle potential added/deleted dirs"; + my $exe_dir = WIN32 ? ".\\" : './'; + system "${exe_dir}perl$Config{_exe}", "-Ilib", "regen/lib_cleanup.pl" + and die "regen/lib_cleanup.pl failed\n"; +} + # # Must clean up, or else t/porting/FindExt.t will fail. # Note that we can always retrieve the original directory with a git checkout. @@ -471,9 +540,10 @@ unlink "cpan/$new_file" unless $tarball; # chdir "t"; say "Running module tests"; -my @test_files = grep { /\.t$/ } find_type_f( $pkg_dir ); -my $exe_dir= $^O =~ /MSWin/ ? "..\\" : './'; -my $output = `${exe_dir}perl$Config{_exe} TEST @test_files`; +my @test_files = grep { /\.t$/ } find_type_f( "../cpan/$pkg_dir" ); +my $exe_dir = WIN32 ? "..\\" : './'; +$ENV{TEST_JOBS} = $_ for grep defined, $make_jobs; +my $output = `${exe_dir}perl$Config{_exe} -I../lib harness @test_files`; unless ($output =~ /All tests successful/) { say $output; exit 1; @@ -494,7 +564,6 @@ print "\n"; say "Failed tests: @failed" if @failed; -say "Attempting to update Maintainers.pl"; chdir '..'; open my $Maintainers_pl, '<', 'Porting/Maintainers.pl'; @@ -525,6 +594,7 @@ while (<$Maintainers_pl>) { } if ($found) { + say "Successfully updated Maintainers.pl"; unlink 'Porting/Maintainers.pl'; rename 'Maintainers.pl' => 'Porting/Maintainers.pl'; chmod 0755 => 'Porting/Maintainers.pl'; @@ -534,8 +604,20 @@ else { say "Make sure you update this by hand before committing."; } -say "$o_module is now version $new_version"; -say "Now you ought to run a make; make test ..."; +print <<"EOF"; + +======================================================================= + +$o_module is now at version $new_version +Next, you should run a "make test". + +Hopefully that will complete successfully, but if not, you can make any +changes you need to get the tests to pass. Don't forget that you'll need +a "CUSTOMIZED" entry in Porting/Maintainers.pl if you change any of the +files under cpan/$pkg_dir. + +Once all tests pass, you can "git add -u" and "git commit" the changes. +EOF __END__ -- Perl5 Master Repository
