RE: [HEADSUP] Upcoming GNU sort removal
Currently the sort structures in the sort utility use 32-bits counters. With Tb files, this may be the limitation point. I can take a look into fixing that. Oleg From: Dennis Glatting [mailto:free...@penx.com] Sent: Thursday, October 04, 2012 8:09 PM To: Gabor Kovesdan Cc: curr...@freebsd.org; Oleg Moskalenko Subject: Re: [HEADSUP] Upcoming GNU sort removal On Thu, 2012-10-04 at 18:16 +0200, Gabor Kovesdan wrote: Em 04-10-2012 16:50, Dennis Glatting escreveu: On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) And did it go much better with base GNU sort? It's quite an extreme case... :) Multi GB is also rare not speaking about multi TB... Yes. However my problem now is ZFS stability -- typically locking up, case example today: last pid: 67998; load averages: 0.00, 0.00, 0.00up 1+19:50:51 19:02:10 80 processes: 1 running, 79 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 146M Active, 2765M Inact, 35G Wired, 371M Buf, 86G Free ARC: 32G Total, 4141M MRU, 27G MFU, 55M Anon, 485M Header, 614M Other Swap: 233G Total, 233G Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 17517 root 17 424 217M 128M tx-tx 21 25.3H 0.00% pbzip2 17568 root 17 524 201M 116M tx-tx 24 25.2H 0.00% pbzip2 17508 root 17 464 201M 116M tx-tx 33 24.6H 0.00% pbzip2 17544 root 17 524 205M 120M tx-tx 37 24.6H 0.00% pbzip2 17532 root 17 524 209M 123M tx-tx 35 24.5H 0.00% pbzip2 etc. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
04.10.2012 13:53, Gabor Kovesdan wrote: it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. http://scan.freebsd.your.org/freebsd-head/usr.bin.grep/2012-01-12-amd64/ /usr/src/usr.bin/grep/regex/xmalloc.c:341:7: warning: Use of memory after it is freed hash_table_del(xmalloc_table, ptr); ^ ~~~ 1 warning generated. /usr/src/usr.bin/grep/regex/tre-fastmatch.c:534:3: warning: Result of 'malloc' is converted to a pointer of type 'unsigned int', which is incompatible with sizeof operand type 'int' FILL_BMGS; ^ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:343:19: note: expanded from macro 'FILL_BMGS' fg-sbmGs = xmalloc(fg-len * sizeof(int)); \ ^ /usr/src/usr.bin/grep/regex/xmalloc.h:69:23: note: expanded from macro 'xmalloc' #define xmalloc(size) malloc(size) ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:537:3: warning: Result of 'malloc' is converted to a pointer of type 'unsigned int', which is incompatible with sizeof operand type 'int' FILL_BMGS_WIDE; ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:359:18: note: expanded from macro 'FILL_BMGS_WIDE' fg-bmGs = xmalloc(fg-wlen * sizeof(int)); \ ^ /usr/src/usr.bin/grep/regex/xmalloc.h:69:23: note: expanded from macro 'xmalloc' #define xmalloc(size) malloc(size) ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:707:3: warning: Memory is never released; potential leak of memory pointed to by 'tmp' STORE_MBS_PAT; ^ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:97:14: note: expanded from macro 'STORE_MBS_PAT' return REG_ESPACE;\ ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:766:3: warning: Result of 'malloc' is converted to a pointer of type 'unsigned int', which is incompatible with sizeof operand type 'int' FILL_BMGS; ^ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:343:19: note: expanded from macro 'FILL_BMGS' fg-sbmGs = xmalloc(fg-len * sizeof(int)); \ ^ /usr/src/usr.bin/grep/regex/xmalloc.h:69:23: note: expanded from macro 'xmalloc' #define xmalloc(size) malloc(size) ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:769:3: warning: Result of 'malloc' is converted to a pointer of type 'unsigned int', which is incompatible with sizeof operand type 'int' FILL_BMGS_WIDE; ^~ /usr/src/usr.bin/grep/regex/tre-fastmatch.c:359:18: note: expanded from macro 'FILL_BMGS_WIDE' fg-bmGs = xmalloc(fg-wlen * sizeof(int)); \ ^ /usr/src/usr.bin/grep/regex/xmalloc.h:69:23: note: expanded from macro 'xmalloc' #define xmalloc(size) malloc(size) ^~ 5 warnings generated. How about fixing http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/167921 ? /usr/bin/grep /tmp/ports/.amd_mnt/faz/host/usr/ports/textproc/docbook-410/work/\\. Segmentation fault (core dumped) uname -a FreeBSD ar1l0u 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #0 r241156M: Wed Oct 3 13:58:16 EEST 2012 arcade@ar1l0u:/usr/obj/usr/src/sys/MINIMAL amd64 gdb /usr/obj/usr/src/usr.bin/grep/grep grep.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd...(no debugging symbols found)... Core was generated by `grep'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.6 Reading symbols from /usr/lib/liblzma.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/liblzma.so.5 Reading symbols from /usr/lib/libbz2.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libbz2.so.4 Reading symbols from /usr/lib/libgnuregex.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgnuregex.so.5 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x004069d2 in tre_compile_fast () (gdb) bt full #0 0x004069d2 in tre_compile_fast () No symbol table info available. #1 0x00404d06 in tre_fastncomp () No symbol table
Re: [HEADSUP] Upcoming GNU sort removal
05.10.2012 13:57, Volodymyr Kostyrko wrote: 04.10.2012 13:53, Gabor Kovesdan wrote: it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. http://scan.freebsd.your.org/freebsd-head/usr.bin.grep/2012-01-12-amd64/ Please excuse me for this, I misread the subject. -- Sphinx of black quartz, judge my vow. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[HEADSUP] Upcoming GNU sort removal
Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
Em 04-10-2012 16:50, Dennis Glatting escreveu: On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) And did it go much better with base GNU sort? It's quite an extreme case... :) Multi GB is also rare not speaking about multi TB... Gabor ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
On Thu, Oct 4, 2012 at 6:16 PM, Gabor Kovesdan ga...@freebsd.org wrote: Em 04-10-2012 16:50, Dennis Glatting escreveu: On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) And did it go much better with base GNU sort? It's quite an extreme case... :) Multi GB is also rare not speaking about multi TB... Yup. Plus nothing prevents us from using GNU sort from ports for those extremely rare cases. AFAICS, BSD sort is great for most of the workloads and is a perfect replacement for GNU sort. Good work! BTW, its good to see BSD-licensed tools gradually replacing GNU tools in base. Though I'd have really preferred to see resources directed towards getting XEN/Dom0 support to FreeBSD/amd64. This really needs some love, IMHO. ;-) Gabor Thanks, -cpghost. -- Cordula's Web. http://www.cordula.ws/ ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
On 10/4/2012 11:26 AM, C. P. Ghost wrote: BTW, its good to see BSD-licensed tools gradually replacing GNU tools in base. Though I'd have really preferred to see resources directed towards getting XEN/Dom0 support to FreeBSD/amd64. This really needs some love, IMHO. ;-) Gabor Thanks, -cpghost. response type=sarcastic Then give it some love yourself! No one is stopping you! :) /response -- Chuck Burns brea...@gmail.com ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [HEADSUP] Upcoming GNU sort removal
On Thu, 2012-10-04 at 18:16 +0200, Gabor Kovesdan wrote: Em 04-10-2012 16:50, Dennis Glatting escreveu: On Thu, 2012-10-04 at 12:53 +0200, Gabor Kovesdan wrote: Hi, it has been more than 3 months ago that BSD sort became default in HEAD and no serious complaints have been raised against it since then so I plan to permanently remove GNU sort from head in the next days. If you have any objection, please raise it now. Initially I had problems with multi TB files (--unique, five to ten files) but I haven't had to do that in two(?) months. I will be getting back to that project in a month or so. It challanges a system's resources. :) And did it go much better with base GNU sort? It's quite an extreme case... :) Multi GB is also rare not speaking about multi TB... Yes. However my problem now is ZFS stability -- typically locking up, case example today: last pid: 67998; load averages: 0.00, 0.00, 0.00up 1+19:50:51 19:02:10 80 processes: 1 running, 79 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 146M Active, 2765M Inact, 35G Wired, 371M Buf, 86G Free ARC: 32G Total, 4141M MRU, 27G MFU, 55M Anon, 485M Header, 614M Other Swap: 233G Total, 233G Free PID USERNAMETHR PRI NICE SIZERES STATE C TIME WCPU COMMAND 17517 root 17 424 217M 128M tx-tx 21 25.3H 0.00% pbzip2 17568 root 17 524 201M 116M tx-tx 24 25.2H 0.00% pbzip2 17508 root 17 464 201M 116M tx-tx 33 24.6H 0.00% pbzip2 17544 root 17 524 205M 120M tx-tx 37 24.6H 0.00% pbzip2 17532 root 17 524 209M 123M tx-tx 35 24.5H 0.00% pbzip2 etc. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
