Re: [PATCH] csplit: new option --suppress-matched
On 03/31/2013 02:19 AM, Assaf Gordon wrote: On 03/30/13 01:08, Pádraig Brady wrote: On 03/28/2013 10:10 PM, Assaf Gordon wrote: Attached is a new option for csplit, suppress-matched, as been mentioned few times before (e.g. http://lists.gnu.org/archive/html/coreutils/2013-02/msg00170.html ). The awkward case here is with integer boundaries and offsets. ... # Adding in the offset, we currently consider the # offset line as the one to suppress, rather than the matched pattern. This was exactly my original understanding of matched - not just the line that matched the regular expression, but the line that matched the specified pattern (i.e. regexp+offset or integer pattern) - and that's the line suppressed. This could be confusing, but at least it's consistent. So more accurately what we're doing is suppressing the boundary line. So less confusingly and more accurately, this option should probably be named/described as: --suppress-boundary Suppress the boundary line from the start of the second and subsequent splits. I'm fine with whichever name you decide. I find matched more natural, and not so confusing, but boundary is just as good. I do think the description is a bit cumbersome (the from the start of the second and subsequent splits part) - it seems more confusing to me than with just omitting it. It's probably one of those cases that a single example of input+output is worth more than a whole paragraph of explanation... OK I stayed with --suppress-matched I've just added the extra boundary description to the texinfo explanation. Note I've removed the -m short option since we try to avoid them for new stuff. Also it gives us the flexibility in future to add a param to --suppress-matched to suppress X lines before/around/after the matched line, which could also be useful. Note I needed to fix array references in the perl test as follows: -push $new_ent, $cmp; +push @$new_ent, $cmp; -push $new_ent, $post; +push @$new_ent, $post; -push $new_ent, $pre; +push @$new_ent, $pre; -push $new_ent, $e; +push @$new_ent, $e; Will push in a while... thanks, Pádraig.
Re: [PATCH] csplit: new option --suppress-matched
Hello, Pádraig Brady wrote, On 04/10/2013 07:49 AM: On 03/28/2013 10:10 PM, Assaf Gordon wrote: Attached is a new option for csplit, suppress-matched, as been mentioned few times before (e.g. http://lists.gnu.org/archive/html/coreutils/2013-02/msg00170.html ). ... Note I've removed the -m short option since we try to avoid them for new stuff. Also it gives us the flexibility in future to add a param to --suppress-matched to suppress X lines before/around/after the matched line, which could also be useful. Ok. good idea. Note I needed to fix array references in the perl test as follows: -push $new_ent, $cmp; +push @$new_ent, $cmp; Sorry about that. Seems like Perl 5.14 and later (which I use on my dev machine) allows unblessed references to functions that take arrays/hashes ( http://perldoc.perl.org/5.14.0/perldelta.html#Syntactical-Enhancements ). I'll have to remember to avoid such backwards-incompatible syntax. Will push in a while... Thanks! -gordon
Re: [PATCH] delaying dd allocation
On 01/23/2013 10:48 AM, Ondrej Oprala wrote:
+ /* Delay buffer allocation if possible. */
+ if ((skip_records OFF_T_MAX / input_blocksize)
+ || 0 skip_via_lseek (input_file, STDIN_FILENO, 0, SEEK_CUR))
+alloc_ibuf ();
- /* Write a sentinel to the slop after the buffer,
- to allow efficient checking for NUL blocks. */
- assert (sizeof (uintptr_t) = OUTPUT_BLOCK_SLOP);
- memset (obuf + output_blocksize, 1, sizeof (uintptr_t));
+ if ((seek_records OFF_T_MAX / output_blocksize)
+ || 0 skip_via_lseek (output_file, STDOUT_FILENO, 0, SEEK_CUR))
+alloc_obuf ();
This doesn't look right.
skip_via_lseek() doesn't support lseek(..., 0, SEEK_CUR)
and may always warn for tape devices?
Also does skip bytes need to be considered?
Also you don't always need to allocate both buffers.
How about avoiding new conditions altogether and just
alloc_[io]buf() before they're needed
(i.e. closer to the read() as I suggested originally).
Also I adjusted the memory limits in the test,
to reduce the chances of false positives,
and bolstered the test cases to cater for the
new allocation calls within skip().
I'll push the attached soon.
thanks,
Pádraig.
From 87c883e00f043faf669040a96a24c59bdeeda662 Mon Sep 17 00:00:00 2001
From: Ondrej Oprala oopr...@redhat.com
Date: Tue, 22 Jan 2013 14:21:23 +0100
Subject: [PATCH] dd: avoid buffer allocations unless needed
* src/dd.c: Add new static global variable ibuf.
(alloc_ibuf, alloc_obuf): New functions factored from dd_copy().
(dd_copy): Call the new functions to allocate memory for
ibuf and obuf when necessary.
(skip): Likewise.
* tests/dd/no-allocate.sh: New test.
* tests/local.mk: Reference the test.
---
src/dd.c| 135 --
tests/dd/no-allocate.sh | 53 ++
tests/local.mk |1 +
3 files changed, 136 insertions(+), 53 deletions(-)
create mode 100755 tests/dd/no-allocate.sh
diff --git a/src/dd.c b/src/dd.c
index c98e578..f727a5e 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -236,6 +236,9 @@ static uintmax_t r_truncate = 0;
static char newline_character = '\n';
static char space_character = ' ';
+/* Input buffer. */
+static char *ibuf;
+
/* Output buffer. */
static char *obuf;
@@ -646,6 +649,65 @@ Options are:\n\
exit (status);
}
+static char *
+human_size (size_t n)
+{
+ static char hbuf[LONGEST_HUMAN_READABLE + 1];
+ int human_opts =
+(human_autoscale | human_round_to_nearest | human_base_1024
+ | human_space_before_unit | human_SI | human_B);
+ return human_readable (n, hbuf, human_opts, 1, 1);
+}
+
+/* Ensure input buffer IBUF is allocated. */
+
+static void
+alloc_ibuf (void)
+{
+ if (ibuf)
+return;
+
+ char *real_buf = malloc (input_blocksize + INPUT_BLOCK_SLOP);
+ if (!real_buf)
+error (EXIT_FAILURE, 0,
+ _(memory exhausted by input buffer of size %zu bytes (%s)),
+ input_blocksize, human_size (input_blocksize));
+
+ real_buf += SWAB_ALIGN_OFFSET; /* allow space for swab */
+
+ ibuf = ptr_align (real_buf, page_size);
+}
+
+/* Ensure output buffer OBUF is allocated/initialized. */
+
+static void
+alloc_obuf (void)
+{
+ if (obuf)
+return;
+
+ if (conversions_mask C_TWOBUFS)
+{
+ /* Page-align the output buffer, too. */
+ char *real_obuf = malloc (output_blocksize + OUTPUT_BLOCK_SLOP);
+ if (!real_obuf)
+error (EXIT_FAILURE, 0,
+ _(memory exhausted by output buffer of size %zu bytes (%s)),
+ output_blocksize, human_size (output_blocksize));
+ obuf = ptr_align (real_obuf, page_size);
+}
+ else
+{
+ alloc_ibuf ();
+ obuf = ibuf;
+}
+
+ /* Write a sentinel to the slop after the buffer,
+ to allow efficient checking for NUL blocks. */
+ assert (sizeof (uintptr_t) = OUTPUT_BLOCK_SLOP);
+ memset (obuf + output_blocksize, 1, sizeof (uintptr_t));
+}
+
static void
translate_charset (char const *new_trans)
{
@@ -1526,7 +1588,7 @@ skip_via_lseek (char const *filename, int fdesc, off_t offset, int whence)
/* Throw away RECORDS blocks of BLOCKSIZE bytes plus BYTES bytes on
file descriptor FDESC, which is open with read permission for FILE.
- Store up to BLOCKSIZE bytes of the data at a time in BUF, if
+ Store up to BLOCKSIZE bytes of the data at a time in IBUF or OBUF, if
necessary. RECORDS or BYTES must be nonzero. If FDESC is
STDIN_FILENO, advance the input offset. Return the number of
records remaining, i.e., that were not skipped because EOF was
@@ -1535,7 +1597,7 @@ skip_via_lseek (char const *filename, int fdesc, off_t offset, int whence)
static uintmax_t
skip (int fdesc, char const *file, uintmax_t records, size_t blocksize,
- size_t *bytes, char *buf)
+ size_t *bytes)
{
uintmax_t offset = records * blocksize + *bytes;
@@ -1607,6 +1669,18 @@ skip (int fdesc, char const *file, uintmax_t records, size_t blocksize,
}
/* else file_size offset
bug#14174: BUG REP: tee takes an annoyingly long time in some system.
Hello sir, Not sure if it is tee's problem. Tee works well in my workstation, but it will take a long time on cluster. See the example below. Could you if possible show me the reason why tee spends so long time while it is working on cluster? This simple tests below illustrate how rep_age can take 30 times longer than necessary: 1. Without tee @3 seconds /home/sqdev16 time echo ' SET TRANSACTION ISOLATION LEVEL READ COMMITTED, READ WRITE, NO ROLLBACK OFF, MULTI COMMIT ON; DELETE WITH MULTI COMMIT FROM manageability.instance_repository.problem_instance_table WHERE (gen_ts_lct) ((CURRENT_TIMESTAMP) - CAST(CAST(0 AS INTEGER) AS INTERVAL HOUR(2))); ' | sqlci --- SQL operation complete. + --- 0 row(s) deleted. End of MXCI Session real0m3.354s user0m0.184s sys 0m0.099s 2. With tee @ 1 minute 46 seconds: /home/sqdev16 time echo ' SET TRANSACTION ISOLATION LEVEL READ COMMITTED, READ WRITE, NO ROLLBACK OFF, MULTI COMMIT ON; DELETE WITH MULTI COMMIT FROM manageability.instance_repository.problem_instance_table WHERE (gen_ts_lct) ((CURRENT_TIMESTAMP) - CAST(CAST(0 AS INTEGER) AS INTERVAL HOUR(2))); ' | sqlci | tee -a junk --- SQL operation complete. + --- 0 row(s) deleted. End of MXCI Session real1m45.969s user0m0.180s sys 0m0.093s Thanks, Jie Gao, Jie (Kyrie) BIP, DS CDC, HPIT *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- Email: jie@hp.com
bug#14024: Test failure in coreutils 8.13
Pádraig, The make check TESTS... failed the same on two runs. (The first performed many compilations, but ended the same.) After this had been run, src/ginstall was present (previously when I failed to find it last time, it must have been removed by make clean). Versions you requested: (Could not find how to get version from /usr/bin/install ! Its man page refers to BSD May 7, 2001, and mentions FreeBSD 4.4.) install --version install: illegal option -- - usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode] [-o owner] file1 file2 install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode] [-o owner] file1 ... fileN directory install -d [-v] [-g group] [-m mode] [-o owner] directory ... /usr/local/bin/install --version install (GNU coreutils) 8.21 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by David MacKenzie. src/ginstall --version install (GNU coreutils) 8.21 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by David MacKenzie. The full stdoutstderr from make check TESTS... was redirected to testinst.log. which is the same as /tests/test-suite.log, except for extra lines at the start and end. This file is included at the end, after *** lines. Thanks, Ellis On 27 Mar 2013, at 21:30, Pádraig Brady wrote: Does that test failure happen every time? I.E. does this fail every time? make check TESTS=tests/install/install-C.sh VERBOSE=yes SUBDIRS=. Can you confirm the versions of install available: install --version /usr/local/bin/install --version src/ginstall --version If src/ginstall is different to the other two, then it would be worth repeating the following with it: *** ** make check TESTS=tests/install/install-C.sh VERBOSE=yes SUBDIRS=. testinst.log *** GENpublic-submodule-commit make check-recursive Making check in . make check-TESTS check-local FAIL: tests/install/install-C.sh GNU coreutils 8.21: ./tests/test-suite.log 1 of 1 test failed. .. contents:: :depth: 2 FAIL: tests/install/install-C.sh (exit: 1) == ++ initial_cwd_=/Gnu/coreutils/coreutils-8.21 ++ fail=0 +++ testdir_prefix_ +++ printf gt ++ pfx_=gt +++ mktempd_ /Gnu/coreutils/coreutils-8.21 gt-install-C.sh. +++ case $# in +++ destdir_=/Gnu/coreutils/coreutils-8.21 +++ template_=gt-install-C.sh. +++ MAX_TRIES_=4 +++ case $destdir_ in +++ case $template_ in unset TMPDIR +++ d=/Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 +++ case $d in +++ test -d /Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 ls -dgo /Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 tr S - +++ perms='drwx-- 2 68 Apr 10 21:27 /Gnu/coreutils/coreutils-8.21/ gt-install-C.sh.DBq8' +++ case $perms in +++ test 0 = 0 +++ echo /Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 +++ return ++ test_dir_=/Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 ++ cd /Gnu/coreutils/coreutils-8.21/gt-install-C.sh.DBq8 ++ gl_init_sh_nl_=' ' ++ IFS=' ' ++ for sig_ in 1 2 3 13 15 +++ expr 1 + 128 ++ eval 'trap '\''Exit 129'\'' 1' +++ trap 'Exit 129' 1 ++ for sig_ in 1 2 3 13 15 +++ expr 2 + 128 ++ eval 'trap '\''Exit 130'\'' 2' +++ trap 'Exit 130' 2 ++ for sig_ in 1 2 3 13 15 +++ expr 3 + 128 ++ eval 'trap '\''Exit 131'\'' 3' +++ trap 'Exit 131' 3 ++ for sig_ in 1 2 3 13 15 +++ expr 13 + 128 ++ eval 'trap '\''Exit 141'\'' 13' +++ trap 'Exit 141' 13 ++ for sig_ in 1 2 3 13 15 +++ expr 15 + 128 ++ eval 'trap '\''Exit 143'\'' 15' +++ trap 'Exit 143' 15 ++ trap remove_tmp_ 0 + path_prepend_ ./src + test 1 '!=' 0 + path_dir_=./src + case $path_dir_ in + abs_path_dir_=/Gnu/coreutils/coreutils-8.21/./src + case $abs_path_dir_ in + PATH=/Gnu/coreutils/coreutils-8.21/./src:/Gnu/coreutils/ coreutils-8.21/src:.:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/ X11/bin:/usr/X11/bin + create_exe_shims_ /Gnu/coreutils/coreutils-8.21/./src + case $EXEEXT in + return 0 + shift + test 0 '!=' 0 + export PATH + print_ver_ ginstall + test yes = yes + local i + for i in '$*' + env ginstall --version install (GNU coreutils) 8.21 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html . This is free software: you are free to change and
bug#14024: Test failure in coreutils 8.13
Could you indicate the file system you're using, and give the output of: echo test a /usr/local/bin/install -Cv -m0644 a b /usr/local/bin/stat a b /usr/local/bin/install -Cv -m0644 a b thanks, Pádraig.
