Hi Lucas, we use Java to run largely our own programs. None do anything special or interesting with the disk, it is simply where we deploy our .jar files and scratch space for e.g. logs
The fragmentation idea is interesting, but it seems unlikely that the disk would be fatally fragmented at ~ 5% utilization We do not have any swap configured, whether on btrfs or otherwise. Java version is 8u25. Thanks, Steven > On Feb 12, 2015, at 4:33 PM, Lucas Smith <vedal...@lksmith.net> wrote: > > Steven: as an avid Java user and now btrfs user, a few things come to mind > but are based on what you're running inside the jvm. What program are you > using Java to run? and I know kvm disk images need to have the +C attribute > to prevent fragmentation of those images. You could try that however, I have > reason to believe your jvm is being weird with regards to the pagefile Aka > swapfile. What version of Java are you running? > end > > On February 12, 2015 5:12:25 AM CST, Steven Schlansker > <sschlans...@opentable.com> wrote: > [ Please CC me on replies, I'm not on the list ] > [ This is a followup to > http://www.spinics.net/lists/linux-btrfs/msg41496.html ] > > Hello linux-btrfs, > I've been having troubles keeping my Apache Mesos / Docker slave nodes > stable. After some period of load, tasks begin to hang. Once this happens > task after task ends up waiting at the same point, never to return. The > system quickly becomes unusable and must be terminated. > > After the previous issues, I was encouraged to upgrade and retry. I am now > running > > Linux 3.19.0 #1 SMP Mon Feb 9 09:43:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux > Btrfs v3.18.2 (and this version was also used to mkfs) > > root@ip-10-30-38-86:~# btrfs fi show > Label: none uuid: 0e8c3f1d-b07b-4643-9834-a41dafb80257 > Total devices 2 FS bytes used 3.92GiB > devid 1 size 74.99GiB used 4.01GiB path /dev/xvdc > > devid 2 size 74.99GiB used 4.01GiB path /dev/xvdd > > > Btrfs v3.18.2 > > Data, RAID0: total=6.00GiB, used=3.69GiB > System, RAID0: total=16.00MiB, used=16.00KiB > Metadata, RAID0: total=2.00GiB, used=229.30MiB > GlobalReserve, single: total=80.00MiB, used=0.00B > > This is the first hung task: > > [146280.252086] INFO: task java:28252 blocked for more than 120 seconds. > [146280.252096] Tainted: G E 3.19.0 #1 > [146280.252098] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [146280.252102] java D ffff8805584df528 0 28252 1400 > 0x00000000 > [146280.252106] ffff8805584df528 ffff880756a24aa0 0000000000014100 > ffff8805584dffd8 > [146280.252108] 0000000000014100 ffff8807567c31c0 ffff880756a24aa0 > ffff8805584df5d0 > [146280.252109] ffff88075a314a00 ffff8805584df5d0 ffff88077c3f8ce8 > 0000000000000002 > [146280.252111] Call Trace: > [146280.252120] > [<ffffffff8194efa0>] ? bit_wait+0x50/0x50 > > [146280.252122] [<ffffffff8194e770>] io_schedule+0xa0/0x130 > [146280.252125] [<ffffffff8194efcc>] bit_wait_io+0x2c/0x50 > [146280.252127] [<ffffffff8194ec05>] __wait_on_bit+0x65/0x90 > [146280.252131] [<ffffffff81169ad7>] wait_on_page_bit+0xc7/0xd0 > [146280.252134] [<ffffffff810b0840>] ? autoremove_wake_function+0x40/0x40 > [146280.252137] [<ffffffff8117d9ed>] shrink_page_list+0x2fd/0xa90 > [146280.252139] [<ffffffff8117e7ad>] shrink_inactive_list+0x1cd/0x590 > [146280.252141] [<ffffffff8117f5b5>] shrink_lruvec+0x5f5/0x810 > [146280.252144] [<ffffffff81086d01>] ? pwq_activate_delayed_work+0x31/0x90 > [146280.252146] [<ffffffff8117f867>] shrink_zone+0x97/0x240 > [146280.252148] [<ffffffff8117fd75>] do_try_to_free_pages+0x155/0x440 > [146280.252150] [<ffffffff81180257>] try_to_free_mem_cgroup_pages+0xa7/0x130 > [146280.252154] [<ffffffff811d2931>] try_charge+0x151/0x620 > [146280.252158] [<ffffffff81815a05>] ? tcp_schedule_loss_probe+0x145/0x1e0 > [146280.252160] [<ffffffff811d6f48>] mem_cgroup_try_charge+0x98/0x110 > [146280.252164] [<ffffffff8170957e>] ? __alloc_skb+0x7e/0x2b0 > [146280.252166] [<ffffffff8116accf>] __add_to_page_cache_locked+0x7f/0x290 > [146280.252169] [<ffffffff8116af28>] add_to_page_cache_lru+0x28/0x80 > [146280.252171] [<ffffffff8116b00f>] pagecache_get_page+0x8f/0x1c0 > [146280.252173] [<ffffffff81952570>] ? _raw_spin_unlock_bh+0x20/0x40 > [146280.252189] [<ffffffffa0045935>] prepare_pages.isra.19+0xc5/0x180 [btrfs] > [146280.252199] [<ffffffffa00464ec>] __btrfs_buffered_write+0x1cc/0x590 > [btrfs] > [146280.252208] [<ffffffffa0049c07>] btrfs_file_write_iter+0x287/0x510 > [btrfs] > [146280.252211] [<ffffffff813f7076>] ? aa_path_perm+0xd6/0x170 > [146280.252214] [<ffffffff811dfd91>] new_sync_write+0x81/0xb0 > [146280.252216] [<ffffffff811e0537>] vfs_write+0xb7/0x1f0 > [146280.252217] [<ffffffff81950636>] ? mutex_lock+0x16/0x37 > [146280.252219] [<ffffffff811e1146>] SyS_write+0x46/0xb0 > [146280.252221] [<ffffffff819529ed>] system_call_fastpath+0x16/0x1b > > Here is a slightly different stacktrace: > > [158880.240245] INFO: task kworker/u16:6:13974 blocked for more than 120 > seconds. > [158880.240249] Tainted: G E 3.19.0 #1 > [158880.240252] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [158880.240254] kworker/u16:6 D ffff88064e7b76c8 0 13974 2 > 0x00000000 > [158880.240259] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1) > [158880.240260] ffff88064e7b76c8 ffff88066f0c18e0 0000000000014100 > ffff88064e7b7fd8 > [158880.240262] 0000000000014100 ffffffff8201e4a0 ffff88066f0c18e0 > ffff88077c3e06e8 > > [158880.240264] ffff88075a214a00 ffff88077c3e06e8 ffff88064e7b7770 > 0000000000000002 > [158880.240266] Call Trace: > [158880.240268] [<ffffffff8194efa0>] ? bit_wait+0x50/0x50 > [158880.240270] [<ffffffff8194e770>] io_schedule+0xa0/0x130 > [158880.240273] [<ffffffff8194efcc>] bit_wait_io+0x2c/0x50 > [158880.240275] [<ffffffff8194ed9b>] __wait_on_bit_lock+0x4b/0xb0 > [158880.240277] [<ffffffff81169f2e>] __lock_page+0xae/0xb0 > [158880.240279] [<ffffffff810b0840>] ? autoremove_wake_function+0x40/0x40 > [158880.240289] [<ffffffffa00501bd>] lock_delalloc_pages+0x13d/0x1d0 [btrfs] > [158880.240299] [<ffffffffa005fc8a>] ? btrfs_map_block+0x1a/0x20 [btrfs] > [158880.240308] [<ffffffffa0050476>] ? > find_delalloc_range.constprop.46+0xa6/0x160 [btrfs] > [158880.240318] [<ffffffffa0052cb3>] find_lock_delalloc_range+0x143/0x1f0 > [btrfs] > [158880.240326] > [<ffffffffa00534e0>] ? end_extent_writepage+0xa0/0xa0 [btrfs] > > [158880.240335] [<ffffffffa0052de1>] writepage_delalloc.isra.32+0x81/0x160 > [btrfs] > [158880.240343] [<ffffffffa0053fab>] __extent_writepage+0xbb/0x2a0 [btrfs] > [158880.240350] [<ffffffffa00544ca>] > extent_write_cache_pages.isra.29.constprop.49+0x33a/0x3f0 [btrfs] > [158880.240359] [<ffffffffa0055f1d>] extent_writepages+0x4d/0x70 [btrfs] > [158880.240368] [<ffffffffa0039090>] ? btrfs_submit_direct+0x7a0/0x7a0 > [btrfs] > [158880.240371] [<ffffffff8109c0a0>] ? default_wake_function+0x10/0x20 > [158880.240378] [<ffffffffa00360a8>] btrfs_writepages+0x28/0x30 [btrfs] > [158880.240380] [<ffffffff81176d2e>] do_writepages+0x1e/0x40 > [158880.240383] [<ffffffff81209400>] __writeback_single_inode+0x40/0x220 > [158880.240385] [<ffffffff81209f13>] writeback_sb_inodes+0x263/0x430 > [158880.240387] [<ffffffff8120a17f>] > __writeback_inodes_wb+0x9f/0xd0 > > [158880.240389] [<ffffffff8120a3f3>] wb_writeback+0x243/0x2c0 > [158880.240392] [<ffffffff8120c9b3>] bdi_writeback_workfn+0x113/0x440 > [158880.240394] [<ffffffff810981bc>] ? finish_task_switch+0x6c/0x1a0 > [158880.240397] [<ffffffff81088f3f>] process_one_work+0x14f/0x3f0 > [158880.240399] [<ffffffff810896a1>] worker_thread+0x121/0x4e0 > [158880.240402] [<ffffffff81089580>] ? rescuer_thread+0x3a0/0x3a0 > [158880.240404] [<ffffffff8108ea72>] kthread+0xd2/0xf0 > [158880.240406] [<ffffffff8108e9a0>] ? kthread_create_on_node+0x180/0x180 > [158880.240408] [<ffffffff8195293c>] ret_from_fork+0x7c/0xb0 > [158880.240411] [<ffffffff8108e9a0>] ? kthread_create_on_node+0x180/0x180 > > > Help! > Thanks, > Steven > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to > majord...@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
