bug#19969: problem: wc -c doesn't read actual # of bytes in file
On 02/03/15 21:29, Linda Walsh wrote: Jim Meyering wrote: As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n Thanks for the report. However, with wc from coreutils-8.23 and a 3.10 kernel, this is no longer an issue. --- with coreutils 8.23 from suse 13.2 and uname: Linux Ishtar 3.18.5-Isht-Van #1 SMP PREEMPT Wed Feb 4 14:50:44 PST 2015 x86_64 x86_64 x86_64 GNU/Linux it is an issue. All the /proc/sys entries are still 0. The issue will be addressed in 8.24: http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=v8.23-47-g2662702 You can build this locally and test just with the src/wc binary without worrying about overwriting any packaged binaries. Pádraig.
bug#19969: problem: wc -c doesn't read actual # of bytes in file
Jim Meyering wrote:
As root:
# cd /proc
# find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f !
-name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c
--files0-from=- |sort -n
Thanks for the report.
However, with wc from coreutils-8.23 and a 3.10 kernel, this is no
longer an issue.
---
with coreutils 8.23 from suse 13.2 and uname:
Linux Ishtar 3.18.5-Isht-Van #1 SMP PREEMPT Wed Feb 4 14:50:44 PST 2015 x86_64
x86_64 x86_64 GNU/Linux
it is an issue. All the /proc/sys entries are still 0.
Here's the output (with a some lines elided)...
0 mpt/summary
0 net/netfilter/nfnetlink_log
0 sys/abi/vsyscall32
0 sys/debug/exception-trace
0 sys/dev/hpet/max-user-freq
0 sys/dev/raid/speed_limit_max
0 sys/dev/raid/speed_limit_min
0 sys/dev/scsi/logging_level
0 sys/fs/aio-max-nr
0 sys/fs/aio-nr
0 sys/fs/dentry-state
0 sys/fs/dir-notify-enable
0 sys/fs/epoll/max_user_watches
0 sys/fs/file-max
0 sys/fs/file-nr
0 sys/fs/inode-nr
0 sys/fs/inode-state
0 sys/fs/inotify/max_queued_events
0 sys/fs/inotify/max_user_instances
0 sys/fs/inotify/max_user_watches
0 sys/fs/lease-break-time
0 sys/fs/leases-enable
0 sys/fs/mqueue/msg_default
0 sys/fs/mqueue/msg_max
0 sys/fs/mqueue/msgsize_default
0 sys/fs/mqueue/msgsize_max
0 sys/fs/mqueue/queues_max
0 sys/fs/nr_open
0 sys/fs/overflowgid
0 sys/fs/overflowuid
0 sys/fs/pipe-max-size
0 sys/fs/protected_hardlinks
0 sys/fs/protected_symlinks
0 sys/fs/suid_dumpable
0 sys/fs/xfs/age_buffer_centisecs
0 sys/fs/xfs/error_level
0 sys/fs/xfs/filestream_centisecs
0 sys/fs/xfs/inherit_noatime
0 sys/fs/xfs/inherit_nodefrag
0 sys/fs/xfs/inherit_nodump
0 sys/fs/xfs/inherit_nosymlinks
0 sys/fs/xfs/inherit_sync
0 sys/fs/xfs/irix_sgid_inherit
0 sys/fs/xfs/irix_symlink_mode
0 sys/fs/xfs/panic_mask
0 sys/fs/xfs/rotorstep
0 sys/fs/xfs/speculative_prealloc_lifetime
0 sys/fs/xfs/stats_clear
0 sys/fs/xfs/xfsbufd_centisecs
0 sys/fs/xfs/xfssyncd_centisecs
0 sys/fscache/object_max_active
0 sys/fscache/operation_max_active
0 sys/kernel/acct
0 sys/kernel/auto_msgmni
0 sys/kernel/bootloader_type
0 sys/kernel/bootloader_version
0 sys/kernel/cad_pid
0 sys/kernel/cap_last_cap
0 sys/kernel/compat-log
0 sys/kernel/core_pattern
0 sys/kernel/core_pipe_limit
0 sys/kernel/core_uses_pid
0 sys/kernel/ctrl-alt-del
0 sys/kernel/dmesg_restrict
0 sys/kernel/domainname
0 sys/kernel/ftrace_dump_on_oops
0 sys/kernel/hostname
0 sys/kernel/hotplug
0 sys/kernel/hung_task_check_count
0 sys/kernel/hung_task_panic
0 sys/kernel/hung_task_timeout_secs
0 sys/kernel/hung_task_warnings
0 sys/kernel/io_delay_type
0 sys/kernel/kexec_load_disabled
0 sys/kernel/keys/gc_delay
0 sys/kernel/keys/maxbytes
0 sys/kernel/keys/maxkeys
0 sys/kernel/keys/persistent_keyring_expiry
0 sys/kernel/keys/root_maxbytes
0 sys/kernel/keys/root_maxkeys
0 sys/kernel/kptr_restrict
0 sys/kernel/kstack_depth_to_print
0 sys/kernel/latencytop
0 sys/kernel/lock_stat
0 sys/kernel/max_lock_depth
0 sys/kernel/modprobe
0 sys/kernel/modules_disabled
0 sys/kernel/msg_next_id
0 sys/kernel/msgmax
0 sys/kernel/msgmnb
0 sys/kernel/msgmni
0 sys/kernel/ngroups_max
0 sys/kernel/nmi_watchdog
0 sys/kernel/ns_last_pid
0 sys/kernel/numa_balancing
0 sys/kernel/numa_balancing_scan_delay_ms
0 sys/kernel/numa_balancing_scan_period_max_ms
0 sys/kernel/numa_balancing_scan_period_min_ms
0 sys/kernel/numa_balancing_scan_size_mb
0 sys/kernel/osrelease
0 sys/kernel/ostype
0 sys/kernel/overflowgid
0 sys/kernel/overflowuid
0 sys/kernel/panic
0 sys/kernel/panic_on_io_nmi
0 sys/kernel/panic_on_oops
0 sys/kernel/panic_on_unrecovered_nmi
0 sys/kernel/perf_cpu_time_max_percent
0 sys/kernel/perf_event_max_sample_rate
0 sys/kernel/perf_event_mlock_kb
0 sys/kernel/perf_event_paranoid
0 sys/kernel/pid_max
0 sys/kernel/poweroff_cmd
0 sys/kernel/print-fatal-signals
0 sys/kernel/printk
0 sys/kernel/printk_delay
0 sys/kernel/printk_ratelimit
0 sys/kernel/printk_ratelimit_burst
0 sys/kernel/pty/max
0 sys/kernel/pty/nr
0 sys/kernel/pty/reserve
0 sys/kernel/random/boot_id
0 sys/kernel/random/entropy_avail
0 sys/kernel/random/poolsize
0 sys/kernel/random/read_wakeup_threshold
0 sys/kernel/random/urandom_min_reseed_secs
0 sys/kernel/random/uuid
0 sys/kernel/random/write_wakeup_threshold
0 sys/kernel/randomize_va_space
0 sys/kernel/sched_autogroup_enabled
0 sys/kernel/sched_cfs_bandwidth_slice_us
0 sys/kernel/sched_child_runs_first
0 sys/kernel/sched_domain/cpu{0..11}/domain0/busy_factor
0 sys/kernel/sched_domain/cpu{0..11}/domain0/busy_idx
0 sys/kernel/sched_domain/cpu{0..11}/domain0/cache_nice_tries
0 sys/kernel/sched_domain/cpu{0..11}/domain0/flags
0 sys/kernel/sched_domain/cpu{0..11}/domain0/forkexec_idx
0 sys/kernel/sched_domain/cpu{0..11}/domain0/idle_idx
0 sys/kernel/sched_domain/cpu{0..11}/domain0/imbalance_pct
0 sys/kernel/sched_domain/cpu{0..11}/domain0/max_interval
0 sys/kernel/sched_domain/cpu{0..11}/domain0/max_newidle_lb_cost
0 sys/kernel/sched_domain/cpu{0..11}/domain0/min_interval
0 sys/kernel/sched_domain/cpu{0..11}/domain0/name
0
bug#19969: problem: wc -c doesn't read actual # of bytes in file
On Mon, Mar 2, 2015 at 1:29 PM, Linda Walsh coreut...@tlinx.org wrote: Jim Meyering wrote: As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n Thanks for the report. However, with wc from coreutils-8.23 and a 3.10 kernel, this is no longer an issue. --- with coreutils 8.23 from suse 13.2 and uname: Linux Ishtar 3.18.5-Isht-Van #1 SMP PREEMPT Wed Feb 4 14:50:44 PST 2015 x86_64 x86_64 x86_64 GNU/Linux it is an issue. All the /proc/sys entries are still 0. Here's the output (with a some lines elided)... 0 mpt/summary 0 net/netfilter/nfnetlink_log Correction: I was using wc from upstream git (coreutils-8.23.141), so as Pádraig says, it will be fixed in 8.24.
bug#19969: problem: wc -c doesn't read actual # of bytes in file
On 02/28/2015 01:59 AM, Linda Walsh wrote: (coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. The Linux kernel is notoriously bad about advertising 0-length file size for non-empty contents of files within sysfs and procfs. Any program that trusts just stat() output will report 0; the only way to see non-zero sizes on these special files is to open() and read() them (I'm not even sure if lseek(SEEK_END) does the trick) - but fixing that is something for the kernel folks to do, and not coreutils to work around, because it is more than coreutils that is affected by the kernel's lies. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
bug#19969: problem: wc -c doesn't read actual # of bytes in file
On 02 Mar 2015 06:57, Eric Blake wrote: On 02/28/2015 01:59 AM, Linda Walsh wrote: (coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. The Linux kernel is notoriously bad about advertising 0-length file size for non-empty contents of files within sysfs and procfs. Any program that trusts just stat() output will report 0; the only way to see non-zero sizes on these special files is to open() and read() them (I'm not even sure if lseek(SEEK_END) does the trick) - but fixing that is something for the kernel folks to do, and not coreutils to work around, because it is more than coreutils that is affected by the kernel's lies. to be fair, for some pseudo files, it's not feasible or possible to report the real size `wc -c /proc/file` should work -mike signature.asc Description: Digital signature
bug#19969: problem: wc -c doesn't read actual # of bytes in file
On Sat, Feb 28, 2015 at 12:59 AM, Linda Walsh coreut...@tlinx.org wrote: (coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. Note -- wc -l doesn't return '0' on the /proc/sys files. As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n Thanks for the report. However, with wc from coreutils-8.23 and a 3.10 kernel, this is no longer an issue.
bug#19969: problem: wc -c doesn't read actual # of bytes in file
Bernhard Voelker wrote: On 02/28/2015 09:59 AM, Linda Walsh wrote: (coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. Note -- wc -l doesn't return '0' on the /proc/sys files. As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n Thanks for the report. However, I'm not 100% sure what the problem is - as you didn't narrow the case down to a certain file, nor did you show us the output and what you expected. --- Yes. I did. Anything under /proc/sys is affected. There is more than 1 'certain file' that is affected. What I would _expect_ is for it to show the actual # of bytes in the file rather than a size of 0. That's why I included the command to duplicate the problem. Are you saying you tried it and it doesn't do the same thing on your system? Without me debugging the source, how much more do you want me to narrow it down? If you change the wc -c = wc -l, you will see non-zero numbers for files under /sys/proc. BTW .. it seems many files under /sys are also affected. As for files under /proc, the 'net' directory seems to work: Are you wanting a list like this: Ishtar:/proc find -H sys net -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n wc: sys/fs/protected_hardlinks: Permission denied wc: sys/fs/protected_symlinks: Permission denied wc: sys/kernel/cad_pid: Permission denied wc: sys/kernel/usermodehelper/bset: Permission denied wc: sys/kernel/usermodehelper/inheritable: Permission denied wc: sys/net/ipv4/route/flush: Permission denied wc: sys/net/ipv4/tcp_fastopen_key: Permission denied wc: sys/vm/compact_memory: Permission denied 0 net/netfilter/nfnetlink_log 0 sys/abi/vsyscall32 0 sys/debug/exception-trace 0 sys/dev/hpet/max-user-freq 0 sys/dev/raid/speed_limit_max 0 sys/dev/raid/speed_limit_min 0 sys/dev/scsi/logging_level 0 sys/fs/aio-max-nr 0 sys/fs/aio-nr 0 sys/fs/dentry-state 0 sys/fs/dir-notify-enable 0 sys/fs/epoll/max_user_watches 0 sys/fs/file-max 0 sys/fs/file-nr 0 sys/fs/inode-nr 0 sys/fs/inode-state 0 sys/fs/inotify/max_queued_events 0 sys/fs/inotify/max_user_instances 0 sys/fs/inotify/max_user_watches 0 sys/fs/lease-break-time ... (pattern is /proc/sys/* = size==0) vs. directory /proc/net: 11 net/ip_tables_names 17 net/ip_tables_targets 36 net/psched 39 net/connector 47 net/mcfilter 54 net/ip_mr_cache 55 net/ip_tables_matches 71 net/ip_mr_vif 128 net/icmp 128 net/rt_cache 128 net/udplite 142 net/sockstat 171 net/ptype 233 net/raw 321 net/netfilter/nf_log 381 net/packet 644 net/dev_mcast 733 net/fib_triestat 822 net/igmp 896 net/route 1062 net/protocols 1090 net/dev 1188 net/softnet_stat 1234 net/arp 1385 net/snmp 1506 net/stat/arp_cache 1678 net/netlink 1718 net/fib_trie 2077 net/stat/rt_cache 2412 net/netstat 4096 net/rt_acct 6528 net/udp 2 net/unix 41550 net/tcp 92581 total --- Sorry to be unclear, but I would expect the numbers under /proc/sys to look something like the numbers under /proc/net. FWIW, the 'prune' and exceptions in the find are to filter out some degenerate cases (like byte counts in /proc/kmem) as well as too much verbosity (skip the processes, including self and thread-self). I also used the -H switch to not skip initial symlinks, but ignore lower ones. For the numbers under /sys, most seem to return a wrong size.a size of 0, but doing a 'cat FILENAME|wc -c' shows the correct number. I didn't give much data about /sys, as getting a test case that would run just for /proc, w/o hanging or looping took too long (like trying to read 'kmsg' takes forever and will basically hang the test case). A quick shot is http://bugs.gnu.org/18621 --- Certainly looks like what I'm seeing. HOWEVER, why do I get numbers in /proc/net but not under /sys? Seems like there is something else going on here. and the corresponding commit (after the latest v8.23 release) http://git.sv.gnu.org/cgit/coreutils.git/commit/?id=2662702b Does this - i.e., a wc(1) built from git - solve your problem? --- It may work *around* the problem, but doesn't explain the inconsistencies between, for example, /proc/sys and /proc/net. If the patch forces a read, there's a good chance it will work, but the inconsistent returns from /proc/sys, /proc/net AND /sys(seems to often show 4K as noted in the other bug report), would seem to indicted a kernel bug somewhere. If they were ALL 0 or all 4K, then I'd say the kernel doesn't return size for either of those interfaces (and that may be as far as that goes and the workaround is
bug#19969: problem: wc -c doesn't read actual # of bytes in file
On 02/28/2015 09:59 AM, Linda Walsh wrote: (coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. Note -- wc -l doesn't return '0' on the /proc/sys files. As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n Thanks for the report. However, I'm not 100% sure what the problem is - as you didn't narrow the case down to a certain file, nor did you show us the output and what you expected. A quick shot is http://bugs.gnu.org/18621 and the corresponding commit (after the latest v8.23 release) http://git.sv.gnu.org/cgit/coreutils.git/commit/?id=2662702b Does this - i.e., a wc(1) built from git - solve your problem? Thanks have a nice day, Berny
bug#19969: problem: wc -c doesn't read actual # of bytes in file
(coreutils-8.21-7.7.7) wc -c(bytes) doesn't seem to reliably read the number of bytes in a file. I was wanting to find out what the largest data-source files in '/proc' and '/sys' (didn't get around to trying /sys, since all the files under /proc/sys return 0 bytes. Note -- wc -l doesn't return '0' on the /proc/sys files. As root: # cd /proc # find -H [^0-9]* -name self -prune -o -name thread-self -prune -o -type f ! -name kmsg ! -name kcore ! -name kpagecount ! -name kpageflags -print0|wc -c --files0-from=- |sort -n
