On Thu, Sep 27, 2018 at 6:47 PM, Dmitry Vyukov <dvyu...@google.com> wrote: > On Thu, Sep 27, 2018 at 12:27 PM, Dmitry Vyukov <dvyu...@google.com> wrote: >> On Thu, Sep 27, 2018 at 1:35 AM, John Johansen >> <john.johan...@canonical.com> wrote: >>> On 09/26/2018 02:22 PM, Daniel Borkmann wrote: >>>> On 09/26/2018 11:09 PM, Tetsuo Handa wrote: >>>>> Hello, Alexei and Daniel. >>>>> >>>>> Can you show us how to run testcases you are testing? >>>> >>>> Sorry for the delay; currently quite backlogged but will definitely take a >>>> look >>>> at these reports. Regarding your question: majority of test cases are in >>>> the >>>> kernel tree under selftests, see tools/testing/selftests/bpf/ . >>>> >>> >>> Its unlikely to be apparmor. I went through the reports and saw nothing that >>> would indicate apparmor involvement, but the primary reason is what is >>> being tested >>> in upstream apparmor atm. >>> >>> The current upstream code does nothing directly with skbuffs. Its >>> possible that the audit code paths (kernel audit does grab skbuffs) >>> could, but there are only a couple cases that would be triggered in >>> the current fuzzing so this seems to be an unlikely source for such a >>> large leak. >> >> >> Ack. There is no direct evidence against apparmor, I am just trying to >> get at least some hooks re the root cause. >> >> From all the weak indirect evidence, I leaning towards skb allocation >> in an infinite loop (or a timer with infinite rate). >> >>>>> On 2018/09/22 22:25, Tetsuo Handa wrote: >>>>>> Hello. >>>>>> >>>>>> syzbot is reporting many lockup problems on bpf.git / bpf-next.git / >>>>>> net.git / net-next.git trees. >>>>>> >>>>>> INFO: rcu detected stall in br_multicast_port_group_expired (2) >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=15c7ad8cf35a07059e8a697a22527e11d294bc94 >>>>>> >>>>>> INFO: rcu detected stall in tun_chr_close >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=6c50618bde03e5a2eefdd0269cf9739c5ebb8270 >>>>>> >>>>>> INFO: rcu detected stall in discover_timer >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=55da031ddb910e58ab9c6853a5784efd94f03b54 >>>>>> >>>>>> INFO: rcu detected stall in ret_from_fork (2) >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=c83129a6683b44b39f5b8864a1325893c9218363 >>>>>> >>>>>> INFO: rcu detected stall in addrconf_rs_timer >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=21c029af65f81488edbc07a10ed20792444711b6 >>>>>> >>>>>> INFO: rcu detected stall in kthread (2) >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=6accd1ed11c31110fed1982f6ad38cc9676477d2 >>>>>> >>>>>> INFO: rcu detected stall in ext4_filemap_fault >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=817e38d20e9ee53390ac361bf0fd2007eaf188af >>>>>> >>>>>> INFO: rcu detected stall in run_timer_softirq (2) >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=f5a230a3ff7822f8d39fddf8485931bd06ae47fe >>>>>> >>>>>> INFO: rcu detected stall in bpf_prog_ADDR >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=fb4911fd0e861171cc55124e209f810a0dd68744 >>>>>> >>>>>> INFO: rcu detected stall in __run_timers (2) >>>>>> >>>>>> https://syzkaller.appspot.com/bug?id=65416569ddc8d2feb8f19066aa761f5a47f7451a >>>>>> >>>>>> The cause of lockup seems to be flood of printk() messages from memory >>>>>> allocation >>>>>> failures, and one of out_of_memory() messages indicates that >>>>>> skbuff_head_cache >>>>>> usage is huge enough to suspect in-kernel memory leaks. >>>>>> >>>>>> [ 1554.547011] skbuff_head_cache 1847887KB 1847887KB >>>>>> >>>>>> Unfortunately, we cannot find from logs what syzbot is trying to do >>>>>> because constant printk() messages is flooding away syzkaller messages. >>>>>> Can you try running your testcases with kmemleak enabled? >>>>>> >>>>> >>>>> On 2018/09/27 2:35, Dmitry Vyukov wrote: >>>>>> I also started suspecting Apparmor. We switched to Apparmor on Aug 30: >>>>>> https://groups.google.com/d/msg/syzkaller-bugs/o73lO4KGh0w/j9pcH2tSBAAJ >>>>>> Now the instances that use SELinux and Smack explicitly contain that >>>>>> in the name, but the rest are Apparmor. >>>>>> Aug 30 roughly matches these assorted "task hung" reports. Perhaps >>>>>> some Apparmor hook leaks a reference to skbs? >>>>> >>>>> Maybe. They have CONFIG_DEFAULT_SECURITY="apparmor". But I'm wondering why >>>>> this problem is not occurring on linux-next.git when this problem is >>>>> occurring >>>>> on bpf.git / bpf-next.git / net.git / net-next.git trees. Is syzbot >>>>> running >>>>> different testcases depending on which git tree is targeted? >>>>> >>> this is another reason that it is doubtful that its apparmor. > > On Thu, Sep 27, 2018 at 2:52 PM, edumazet >> Have you tried kmemleak perhaps, it might give us a clue, but it seems >> obvious the leak would be in TX path. > > So, I've tried. Now what? :) > > I've uploaded all reports to: > https://drive.google.com/file/d/107LUW0zmYbXmxfQCWoLpeenxXJsXIkxj/view?usp=sharing > This is on net tree d4ce58082f206bf6e7d697380c7bc5480a8b0264 > > memory leak in __lookup_hash 33 Sep 27 2018 16:35:50 > memory leak in new_inode_pseudo 43 Sep 27 2018 16:41:14 > memory leak in path_openat 1 Sep 27 2018 16:12:53 > memory leak in rhashtable_init 1 Sep 27 2018 16:41:41 > memory leak in shmem_symlink 4 Sep 27 2018 16:34:34 > memory leak in __anon_vma_prepare 1 Sep 27 2018 17:30:02 > memory leak in __do_execve_file 4 Sep 27 2018 18:14:10 > memory leak in __do_sys_perf_event_open 2 Sep 27 2018 17:40:05 > memory leak in __es_insert_extent 3 Sep 27 2018 17:24:52 > memory leak in __getblk_gfp 3 Sep 27 2018 18:19:30 > memory leak in __handle_mm_fault 1 Sep 27 2018 18:11:31 > memory leak in __hw_addr_create_ex 2 Sep 27 2018 18:10:56 > memory leak in __ip_mc_inc_group 13 Sep 27 2018 18:24:31 > memory leak in __khugepaged_enter 1 Sep 27 2018 15:40:25 > memory leak in __list_lru_init 27 Sep 27 2018 17:30:48 > memory leak in __neigh_create 12 Sep 27 2018 17:40:28 > memory leak in __netlink_create 1 Sep 27 2018 15:40:23 > memory leak in __register_sysctl_table 1 Sep 27 2018 17:36:57 > memory leak in __send_signal 1 Sep 27 2018 18:30:48 > memory leak in __sys_socket 7 Sep 27 2018 15:43:20 > memory leak in anon_inode_getfile 4 Sep 27 2018 18:17:29 > memory leak in bpf_prog_store_orig_filter 1 Sep 27 2018 17:59:14 > memory leak in br_multicast_new_group 2 Sep 27 2018 18:16:42 > memory leak in br_multicast_new_port_group 3 Sep 27 2018 18:17:39 > memory leak in build_sched_domains 2 Sep 27 2018 17:28:55 > memory leak in clone_mnt 2 Sep 27 2018 18:18:48 > memory leak in compute_effective_progs 1 Sep 27 2018 18:35:15 > memory leak in create_empty_buffers 1 Sep 27 2018 17:35:42 > memory leak in create_filter_start 2 Sep 27 2018 18:16:00 > memory leak in create_pipe_files 3 Sep 27 2018 18:21:47 > memory leak in do_ip6t_set_ctl 1 Sep 27 2018 15:40:21 > memory leak in do_ipt_set_ctl 1 Sep 27 2018 17:37:35 > memory leak in do_signalfd4 1 Sep 27 2018 18:00:45 > memory leak in do_syslog 2 Sep 27 2018 18:06:22 > memory leak in ep_insert 11 Sep 27 2018 17:39:14 > memory leak in ep_ptable_queue_proc 3 Sep 27 2018 17:37:17 > memory leak in ext4_mb_new_group_pa 1 Sep 27 2018 17:36:04 > memory leak in ext4_mb_new_inode_pa 3 Sep 27 2018 17:39:19 > memory leak in fdb_create 13 Sep 27 2018 18:24:03 > memory leak in fib6_add_1 11 Sep 27 2018 18:19:53 > memory leak in fib_table_insert 1 Sep 27 2018 17:36:49 > memory leak in find_get_context 1 Sep 27 2018 15:42:00 > memory leak in fsnotify_add_mark_locked 11 Sep 27 2018 17:31:41 > memory leak in idr_get_free 2 Sep 27 2018 18:17:34 > memory leak in iget_locked 1 Sep 27 2018 17:52:59 > memory leak in inet_frag_find 3 Sep 27 2018 18:23:53 > memory leak in inotify_update_watch 25 Sep 27 2018 17:30:59 > memory leak in ioc_create_icq 1 Sep 27 2018 15:42:40 > memory leak in ip6_pol_route 4 Sep 27 2018 18:24:59 > memory leak in ip6_route_info_create 1 Sep 27 2018 17:38:09 > memory leak in ip6t_register_table 1 Sep 27 2018 17:35:39 > memory leak in ip_route_output_key_hash_rcu 4 Sep 27 2018 18:21:21 > memory leak in ipt_register_table 3 Sep 27 2018 17:39:58 > memory leak in ipv6_add_addr 1 Sep 27 2018 17:38:49 > memory leak in load_elf_binary 1 Sep 27 2018 18:09:44 > memory leak in map_create 9 Sep 27 2018 18:25:10 > memory leak in memcg_update_all_list_lrus 1 Sep 27 2018 15:39:36 > memory leak in ndisc_send_rs 1 Sep 27 2018 17:50:03 > memory leak in neigh_table_init 6 Sep 27 2018 17:23:41 > memory leak in nf_hook_entries_grow 5 Sep 27 2018 15:43:41 > memory leak in packet_sendmsg 1 Sep 27 2018 18:08:43 > memory leak in pcpu_create_chunk 1 Sep 27 2018 17:48:41 > memory leak in prepare_creds 18 Sep 27 2018 17:29:10 > memory leak in prepare_kernel_cred 15 Sep 27 2018 18:23:42 > memory leak in process_preds 2 Sep 27 2018 18:11:01 > memory leak in rht_deferred_worker 9 Sep 27 2018 18:24:29 > memory leak in sched_init_domains 2 Sep 27 2018 18:12:42 > memory leak in sctp_addr_wq_mgmt 1 Sep 27 2018 17:46:20 > memory leak in sget 5 Sep 27 2018 18:03:40 > memory leak in shmem_symlink 30 Sep 27 2018 17:32:09 > memory leak in skb_clone 3 Sep 27 2018 18:13:22 > memory leak in submit_bh_wbc 1 Sep 27 2018 17:49:06 > memory leak in tracepoint_probe_register_prio 1 Sep 27 2018 17:39:13 > memory leak in xt_replace_table 4 Sep 27 2018 15:43:19 > memory leak in __delayacct_tsk_init 2 Sep 27 2018 17:02:53 > memory leak in disk_expand_part_tbl 1 Sep 27 2018 16:59:05 > memory leak in do_ip6t_set_ctl 14 Sep 27 2018 15:46:37 > memory leak in neigh_table_init 4 Sep 27 2018 17:10:29 > memory leak in do_check 1 Sep 27 2018 18:38:13
I see at least 3 bridge-related: memory leak in skb_clone memory leak in br_multicast_new_group memory leak in br_multicast_new_port_group
