https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224160
Bug ID: 224160
Summary: wc -c is slow
Product: Base System
Version: CURRENT
Hardware: Any
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: bin
Assignee: [email protected]
Reporter: [email protected]
The wc(1) command has several optimizations to run as fast as possible.
However, it is still slow in some use cases, much slower than the GNU wc
command
Using the OpenStreetMap database dump planet-latest.osm.bz2
(from https://wiki.openstreetmap.org/wiki/Planet.osm)
which it is a 61GB bzip'd XML file.
I checked how large the uncompressed XML is, on a 32 CPU machine:
# FreeBSD wc
$ pbzip2 -dc planet-latest.osm.bz2 | time wc -c
908171295050
4729.53 real 4400.69 user 199.34 sys
the wc(1) command was running at 100% CPU time, and pbzip2 was using only 500%
CPU time.
I run the tests again with GNU wc. The wc command was using only 20% CPU time,
and pbzip2 around 3000%.
# GNU wc
$ pbzip2 -dc planet-latest.osm.bz2 | time gwc -c
908171295050
2003.15 real 8.86 user 355.53 sys
The FreeBSD wc(1) command is using 500 times more user time (4400 <-> 9) than
the GNU wc, and a little bit less system time (199 <-> 355). The bottleneck was
not pbzip2, it was wc.
We should check why the optimization for wc -c for reading from stdin is not
working.
--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[email protected]"
