Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 19:19:59, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 6:12 PM, Michal Hocko wrote: > > On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: > > [...] > >> > But if the allocation is stuck then the holder of the lock cannot make > >> > a forward progress and it is

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 19:19:59, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 6:12 PM, Michal Hocko wrote: > > On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: > > [...] > >> > But if the allocation is stuck then the holder of the lock cannot make > >> > a forward progress and it is effectivelly

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 6:12 PM, Michal Hocko wrote: > On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: > [...] >> > But if the allocation is stuck then the holder of the lock cannot make >> > a forward progress and it is effectivelly deadlocked because other IO >> > depends on

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 6:12 PM, Michal Hocko wrote: > On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: > [...] >> > But if the allocation is stuck then the holder of the lock cannot make >> > a forward progress and it is effectivelly deadlocked because other IO >> > depends on the lock it holds.

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: [...] > > But if the allocation is stuck then the holder of the lock cannot make > > a forward progress and it is effectivelly deadlocked because other IO > > depends on the lock it holds. Maybe I just ask bad questions but what > > Only I/O to the

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 17:06:51, Ilya Dryomov wrote: [...] > > But if the allocation is stuck then the holder of the lock cannot make > > a forward progress and it is effectivelly deadlocked because other IO > > depends on the lock it holds. Maybe I just ask bad questions but what > > Only I/O to the

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 4:36 PM, Michal Hocko wrote: > On Thu 30-03-17 15:48:42, Ilya Dryomov wrote: >> On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: > [...] >> > familiar with Ceph at all but does any of its (slab) shrinkers generate >> > IO to

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 4:36 PM, Michal Hocko wrote: > On Thu 30-03-17 15:48:42, Ilya Dryomov wrote: >> On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: > [...] >> > familiar with Ceph at all but does any of its (slab) shrinkers generate >> > IO to recurse back? >> >> We don't register any

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 15:48:42, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: [...] > > familiar with Ceph at all but does any of its (slab) shrinkers generate > > IO to recurse back? > > We don't register any custom shrinkers. This is XFS on top of

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 15:48:42, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: [...] > > familiar with Ceph at all but does any of its (slab) shrinkers generate > > IO to recurse back? > > We don't register any custom shrinkers. This is XFS on top of rbd, > a ceph-backed

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 15:53:35, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: [...] > >> are you saying it's OK for a block > >> device to recurse back into the filesystem when doing I/O, potentially > >>

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 15:53:35, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: [...] > >> are you saying it's OK for a block > >> device to recurse back into the filesystem when doing I/O, potentially > >> generating more

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: >> > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> >> On Wed, Mar 29, 2017 at 12:55 PM, Michal

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: >> > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: >> >> > On Wed 29-03-17

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: > On Thu 30-03-17 12:02:03, Ilya Dryomov wrote: >> On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: >> > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: > [...] >> >> We got rid of osdc->request_mutex in

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 1:21 PM, Michal Hocko wrote: > On Thu 30-03-17 12:02:03, Ilya Dryomov wrote: >> On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: >> > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: > [...] >> >> We got rid of osdc->request_mutex in 4.7, so these workers are almost >>

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 12:02:03, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: [...] > >> We got rid of osdc->request_mutex in 4.7, so these workers are almost > >> independent in newer kernels and should

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu 30-03-17 12:02:03, Ilya Dryomov wrote: > On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: [...] > >> We got rid of osdc->request_mutex in 4.7, so these workers are almost > >> independent in newer kernels and should be able to free up

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: >> > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> >> On Wed, Mar 29, 2017 at 12:55 PM, Michal

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Thu, Mar 30, 2017 at 8:25 AM, Michal Hocko wrote: > On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: >> > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: >> >> > On Wed 29-03-17

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: > > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: > >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > >> > On Wed 29-03-17 12:41:26, Michal Hocko

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 16:25:18, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: > > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: > >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > >> > On Wed 29-03-17 12:41:26, Michal Hocko wrote: > >> > [...] > >> >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:49 PM, Brian Foster wrote: > On Wed, Mar 29, 2017 at 01:18:34PM +0200, Michal Hocko wrote: >> On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: >> > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: >> > > On Wed, Mar 29, 2017 at

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:49 PM, Brian Foster wrote: > On Wed, Mar 29, 2017 at 01:18:34PM +0200, Michal Hocko wrote: >> On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: >> > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: >> > > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: >>

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: >> > On Wed 29-03-17 12:41:26, Michal Hocko wrote: >> > [...] >> >> > ceph_con_workfn >> >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:16 PM, Michal Hocko wrote: > On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: >> On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: >> > On Wed 29-03-17 12:41:26, Michal Hocko wrote: >> > [...] >> >> > ceph_con_workfn >> >> > mutex_lock(>mutex) #

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 01:18:34PM +0200, Michal Hocko wrote: > On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: > > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > > > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > > >> [CC xfs guys] > > >> > > >> On Wed

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 01:18:34PM +0200, Michal Hocko wrote: > On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: > > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > > > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > > >> [CC xfs guys] > > >> > > >> On Wed 29-03-17 11:21:44,

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > >> [CC xfs guys] > >> > >> On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: > >> [...] > >> > This is a set of

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 13:14:42, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > >> [CC xfs guys] > >> > >> On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: > >> [...] > >> > This is a set of stack traces from

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: >> [CC xfs guys] >> >> On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: >> [...] >> > This is a set of stack traces from http://tracker.ceph.com/issues/19309 >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 1:05 PM, Brian Foster wrote: > On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: >> [CC xfs guys] >> >> On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: >> [...] >> > This is a set of stack traces from http://tracker.ceph.com/issues/19309 >> > (linked in the

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > > On Wed 29-03-17 12:41:26, Michal Hocko wrote: > > [...] > >> > ceph_con_workfn > >> > mutex_lock(>mutex) # ceph_connection::mutex > >> > try_write > >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 13:10:01, Ilya Dryomov wrote: > On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > > On Wed 29-03-17 12:41:26, Michal Hocko wrote: > > [...] > >> > ceph_con_workfn > >> > mutex_lock(>mutex) # ceph_connection::mutex > >> > try_write > >> > ceph_tcp_connect > >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > On Wed 29-03-17 12:41:26, Michal Hocko wrote: > [...] >> > ceph_con_workfn >> > mutex_lock(>mutex) # ceph_connection::mutex >> > try_write >> > ceph_tcp_connect >> > sock_create_kern >> > GFP_KERNEL

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 12:55 PM, Michal Hocko wrote: > On Wed 29-03-17 12:41:26, Michal Hocko wrote: > [...] >> > ceph_con_workfn >> > mutex_lock(>mutex) # ceph_connection::mutex >> > try_write >> > ceph_tcp_connect >> > sock_create_kern >> > GFP_KERNEL allocation >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > [CC xfs guys] > > On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: > [...] > > This is a set of stack traces from http://tracker.ceph.com/issues/19309 > > (linked in the changelog): > > > > Workqueue: ceph-msgr con_work [libceph] > >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed, Mar 29, 2017 at 12:41:26PM +0200, Michal Hocko wrote: > [CC xfs guys] > > On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: > [...] > > This is a set of stack traces from http://tracker.ceph.com/issues/19309 > > (linked in the changelog): > > > > Workqueue: ceph-msgr con_work [libceph] > >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 12:41:26, Michal Hocko wrote: [...] > > ceph_con_workfn > > mutex_lock(>mutex) # ceph_connection::mutex > > try_write > > ceph_tcp_connect > > sock_create_kern > > GFP_KERNEL allocation > > allocator recurses into XFS, more I/O is issued One more

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Wed 29-03-17 12:41:26, Michal Hocko wrote: [...] > > ceph_con_workfn > > mutex_lock(>mutex) # ceph_connection::mutex > > try_write > > ceph_tcp_connect > > sock_create_kern > > GFP_KERNEL allocation > > allocator recurses into XFS, more I/O is issued One more

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

[CC xfs guys] On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: [...] > This is a set of stack traces from http://tracker.ceph.com/issues/19309 > (linked in the changelog): > > Workqueue: ceph-msgr con_work [libceph] > 8810871cb018 0046 881085d4 >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

[CC xfs guys] On Wed 29-03-17 11:21:44, Ilya Dryomov wrote: [...] > This is a set of stack traces from http://tracker.ceph.com/issues/19309 > (linked in the changelog): > > Workqueue: ceph-msgr con_work [libceph] > 8810871cb018 0046 881085d4 >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue, Mar 28, 2017 at 3:30 PM, Michal Hocko wrote: > On Tue 28-03-17 15:23:58, Ilya Dryomov wrote: >> On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: >> > On Tue 28-03-17 14:30:45, Greg KH wrote: >> >> 4.4-stable review patch. If anyone has any

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue, Mar 28, 2017 at 3:30 PM, Michal Hocko wrote: > On Tue 28-03-17 15:23:58, Ilya Dryomov wrote: >> On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: >> > On Tue 28-03-17 14:30:45, Greg KH wrote: >> >> 4.4-stable review patch. If anyone has any objections, please let me >> >> know. >> >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue 28-03-17 15:23:58, Ilya Dryomov wrote: > On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: > > On Tue 28-03-17 14:30:45, Greg KH wrote: > >> 4.4-stable review patch. If anyone has any objections, please let me know. > > > > I haven't seen the original patch but the

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue 28-03-17 15:23:58, Ilya Dryomov wrote: > On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: > > On Tue 28-03-17 14:30:45, Greg KH wrote: > >> 4.4-stable review patch. If anyone has any objections, please let me know. > > > > I haven't seen the original patch but the changelog makes me

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: > On Tue 28-03-17 14:30:45, Greg KH wrote: >> 4.4-stable review patch. If anyone has any objections, please let me know. > > I haven't seen the original patch but the changelog makes me worried. > How exactly this is a

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue, Mar 28, 2017 at 2:43 PM, Michal Hocko wrote: > On Tue 28-03-17 14:30:45, Greg KH wrote: >> 4.4-stable review patch. If anyone has any objections, please let me know. > > I haven't seen the original patch but the changelog makes me worried. > How exactly this is a problem? Where do we

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue 28-03-17 14:30:45, Greg KH wrote: > 4.4-stable review patch. If anyone has any objections, please let me know. I haven't seen the original patch but the changelog makes me worried. How exactly this is a problem? Where do we lockup? Does rbd/libceph take any xfs locks? >

Re: [PATCH 4.4 48/76] libceph: force GFP_NOIO for socket allocations

On Tue 28-03-17 14:30:45, Greg KH wrote: > 4.4-stable review patch. If anyone has any objections, please let me know. I haven't seen the original patch but the changelog makes me worried. How exactly this is a problem? Where do we lockup? Does rbd/libceph take any xfs locks? >