[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 02:57:25PM -0700, Hugh Dickins wrote:
> On Mon, 7 May 2012, Stephane Marchesin wrote:
> 
> > While investing some Sandy Bridge rendering corruption, I found out
> > that all physical memory pages below 1MiB were returning garbage when
> > read through the GTT. This has been causing graphics corruption (when
> > it's used for textures, render targets and pixmaps) and GPU hangups
> > (when it's used for GPU batch buffers).
> > 
> > I talked with some people at Intel and they confirmed my findings,
> > and said that a couple of other random pages were also affected.
> > 
> > We could fix this problem by adding an e820 region preventing the
> > memory below 1 MiB to be used, but that prevents at least my machine
> > from booting. One could think that we should be able to fix it in
> > i915, but since the allocation is done by the backing shmem this is
> > not possible.
> > 
> > In the end, I came up with the ugly workaround of just leaking the
> > offending pages in shmem.c. I do realize it's truly ugly, but I'm
> > looking for a fix to the existing code, and am wondering if people on
> > this list have a better idea, short of rewriting i915_gem.c to
> > allocate its own pages directly.
> > 
> > Signed-off-by: Stephane Marchesin 
> 
> Well done for discovering and pursuing this issue, but of course (as
> you know: you're trying to provoke us to better) your patch is revolting.
> 
> And not even enough: swapin readahead and swapoff can read back
> from swap into pages which the i915 will later turn out to dislike.
> 
> I do have a shmem.c patch coming up for gma500, which cannot use pages
> over 4GB; but that fits more reasonably with memory allocation policies,
> where we expect that anyone who can use a high page can use a lower as
> well, and there's already __GFP_DMA32 to set the limit.
> 
> Your limitation is at the opposite end, so that patch won't help you at
> all.  And I don't see how Andi's ZONE_DMA exclusion would work, without
> equal hackery to enable private zonelists, avoiding that convention.
> 
> i915 is not the only user of shmem, and x86 not the only architecture:
> we're not going to make everyone suffer for this.  Once the memory
> allocator gets down to giving you the low 1MB, my guess is that it's
> already short of memory, and liable to deadlock or OOM if you refuse
> and soak up every page it then gives you.  Even if i915 has to live
> with that possibility, we're not going to extend it to everyone else.
> 
> arch/x86/Kconfig has X86_RESERVE_LOW, default 64, range 4 640 (and
> I think we reserve all the memory range from 640kB to 1MB anyway).
> Would setting that to 640 allow you to boot, and avoid the i915
> problem on all but the odd five pages?  I'm not pretending that's
> an ideal solution at all (unless freeing initmem could release most
> of it on non-SandyBridge and non-i915 machines), but it would be
> useful to know if that does provide a stopgap solution.  If that
> does work, maybe we just mark the odd five PageReserved at startup.

Hm, as a stopgap measure to make Sandybridge gpus not die that sounds
pretty good. But we still need a more generic solution for the long-term,
see below

> Is there really no way this can be handled closer to the source of
> the problem, in the i915 driver itself?  I do not know the flow of
> control in i915 (and X) at all, but on the surface it seems that the
> problem only comes when you map these problematic pages into the GTT
> (if I'm even using the right terminology), and something (not shmem.c)
> actively has to do that.
> 
> Can't you check the pfn at that point, and if it's an unsuitable page,
> copy into a suitable page (perhaps allocated then, perhaps from a pool
> you primed earlier) and map that suitable page into the GTT instead?
> Maybe using page->private to link them if that helps.
> 
> So long as the page (or its shadow) is mapped into the GTT, I imagine
> it would be pinned, and not liable to be swapped out or otherwise
> interfered with by shmem.c.  And when you unmap it from GTT, copy
> back to the unsuitable shmem object page before unpinning.
> 
> I fully accept that I have very little understanding of GPU DRM GTT
> and i915, and this may be impossible or incoherent: but please, let's
> try to keep the strangeness where it belongs.  If necessary, we'll
> have add some kind of flag and callback from shmem.c to the driver;
> but I'd so much prefer to avoid that.

The copy stuff back approach is pretty much what ttm uses atm: It
allocates suitable pages with whatever means it has (usually through the
dma api) and if the shrinker callback tells it that it's sitting on too
much memory, it copies stuff out to the shmem backing storage used by gem.

There are quite a few issues with that approach:
- We expose mmap to the shmem file directly to userspace in i915. We use
  these extensively on Sandybridge because there direct cpu access is
  coherent with what the gpu does. Original userspace would always 

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Hugh Dickins]

On Mon, 7 May 2012, Stephane Marchesin wrote:

 While investing some Sandy Bridge rendering corruption, I found out
 that all physical memory pages below 1MiB were returning garbage when
 read through the GTT. This has been causing graphics corruption (when
 it's used for textures, render targets and pixmaps) and GPU hangups
 (when it's used for GPU batch buffers).
 
 I talked with some people at Intel and they confirmed my findings,
 and said that a couple of other random pages were also affected.
 
 We could fix this problem by adding an e820 region preventing the
 memory below 1 MiB to be used, but that prevents at least my machine
 from booting. One could think that we should be able to fix it in
 i915, but since the allocation is done by the backing shmem this is
 not possible.
 
 In the end, I came up with the ugly workaround of just leaking the
 offending pages in shmem.c. I do realize it's truly ugly, but I'm
 looking for a fix to the existing code, and am wondering if people on
 this list have a better idea, short of rewriting i915_gem.c to
 allocate its own pages directly.
 
 Signed-off-by: Stephane Marchesin marc...@chromium.org

Well done for discovering and pursuing this issue, but of course (as
you know: you're trying to provoke us to better) your patch is revolting.

And not even enough: swapin readahead and swapoff can read back
from swap into pages which the i915 will later turn out to dislike.

I do have a shmem.c patch coming up for gma500, which cannot use pages
over 4GB; but that fits more reasonably with memory allocation policies,
where we expect that anyone who can use a high page can use a lower as
well, and there's already __GFP_DMA32 to set the limit.

Your limitation is at the opposite end, so that patch won't help you at
all.  And I don't see how Andi's ZONE_DMA exclusion would work, without
equal hackery to enable private zonelists, avoiding that convention.

i915 is not the only user of shmem, and x86 not the only architecture:
we're not going to make everyone suffer for this.  Once the memory
allocator gets down to giving you the low 1MB, my guess is that it's
already short of memory, and liable to deadlock or OOM if you refuse
and soak up every page it then gives you.  Even if i915 has to live
with that possibility, we're not going to extend it to everyone else.

arch/x86/Kconfig has X86_RESERVE_LOW, default 64, range 4 640 (and
I think we reserve all the memory range from 640kB to 1MB anyway).
Would setting that to 640 allow you to boot, and avoid the i915
problem on all but the odd five pages?  I'm not pretending that's
an ideal solution at all (unless freeing initmem could release most
of it on non-SandyBridge and non-i915 machines), but it would be
useful to know if that does provide a stopgap solution.  If that
does work, maybe we just mark the odd five PageReserved at startup.

Is there really no way this can be handled closer to the source of
the problem, in the i915 driver itself?  I do not know the flow of
control in i915 (and X) at all, but on the surface it seems that the
problem only comes when you map these problematic pages into the GTT
(if I'm even using the right terminology), and something (not shmem.c)
actively has to do that.

Can't you check the pfn at that point, and if it's an unsuitable page,
copy into a suitable page (perhaps allocated then, perhaps from a pool
you primed earlier) and map that suitable page into the GTT instead?
Maybe using page-private to link them if that helps.

So long as the page (or its shadow) is mapped into the GTT, I imagine
it would be pinned, and not liable to be swapped out or otherwise
interfered with by shmem.c.  And when you unmap it from GTT, copy
back to the unsuitable shmem object page before unpinning.

I fully accept that I have very little understanding of GPU DRM GTT
and i915, and this may be impossible or incoherent: but please, let's
try to keep the strangeness where it belongs.  If necessary, we'll
have add some kind of flag and callback from shmem.c to the driver;
but I'd so much prefer to avoid that.

Hugh

 
 Change-Id: I957e125fb280e0b0d6b05a83cc4068df2f05aa0a
 ---
  mm/shmem.c |   39 +--
  1 files changed, 37 insertions(+), 2 deletions(-)
 
 diff --git a/mm/shmem.c b/mm/shmem.c
 index 6c253f7..dcbb58b 100644
 --- a/mm/shmem.c
 +++ b/mm/shmem.c
 @@ -768,6 +768,31 @@ redirty:
   return 0;
  }
  
 +/*
 + * Some intel GPUs can't use those pages in the GTT, which results in
 + * graphics corruption. Sadly, it's impossible to prevent usage of those
 + * pages in the intel allocator.
 + *
 + * Instead, we test for those areas here and leak the corresponding pages.
 + *
 + * Some day, when the intel GPU memory is not backed by shmem any more,
 + * we'll be able to come up with a solution which is contained in i915.
 + */
 +static bool i915_usable_page(struct page *page)
 +{
 + dma_addr_t addr = page_to_phys(page);
 +
 + if (unlikely((addr  1 * 1024 * 1024) ||
 +   

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 02:57:25PM -0700, Hugh Dickins wrote:
 On Mon, 7 May 2012, Stephane Marchesin wrote:
 
  While investing some Sandy Bridge rendering corruption, I found out
  that all physical memory pages below 1MiB were returning garbage when
  read through the GTT. This has been causing graphics corruption (when
  it's used for textures, render targets and pixmaps) and GPU hangups
  (when it's used for GPU batch buffers).
  
  I talked with some people at Intel and they confirmed my findings,
  and said that a couple of other random pages were also affected.
  
  We could fix this problem by adding an e820 region preventing the
  memory below 1 MiB to be used, but that prevents at least my machine
  from booting. One could think that we should be able to fix it in
  i915, but since the allocation is done by the backing shmem this is
  not possible.
  
  In the end, I came up with the ugly workaround of just leaking the
  offending pages in shmem.c. I do realize it's truly ugly, but I'm
  looking for a fix to the existing code, and am wondering if people on
  this list have a better idea, short of rewriting i915_gem.c to
  allocate its own pages directly.
  
  Signed-off-by: Stephane Marchesin marc...@chromium.org
 
 Well done for discovering and pursuing this issue, but of course (as
 you know: you're trying to provoke us to better) your patch is revolting.
 
 And not even enough: swapin readahead and swapoff can read back
 from swap into pages which the i915 will later turn out to dislike.
 
 I do have a shmem.c patch coming up for gma500, which cannot use pages
 over 4GB; but that fits more reasonably with memory allocation policies,
 where we expect that anyone who can use a high page can use a lower as
 well, and there's already __GFP_DMA32 to set the limit.
 
 Your limitation is at the opposite end, so that patch won't help you at
 all.  And I don't see how Andi's ZONE_DMA exclusion would work, without
 equal hackery to enable private zonelists, avoiding that convention.
 
 i915 is not the only user of shmem, and x86 not the only architecture:
 we're not going to make everyone suffer for this.  Once the memory
 allocator gets down to giving you the low 1MB, my guess is that it's
 already short of memory, and liable to deadlock or OOM if you refuse
 and soak up every page it then gives you.  Even if i915 has to live
 with that possibility, we're not going to extend it to everyone else.
 
 arch/x86/Kconfig has X86_RESERVE_LOW, default 64, range 4 640 (and
 I think we reserve all the memory range from 640kB to 1MB anyway).
 Would setting that to 640 allow you to boot, and avoid the i915
 problem on all but the odd five pages?  I'm not pretending that's
 an ideal solution at all (unless freeing initmem could release most
 of it on non-SandyBridge and non-i915 machines), but it would be
 useful to know if that does provide a stopgap solution.  If that
 does work, maybe we just mark the odd five PageReserved at startup.

Hm, as a stopgap measure to make Sandybridge gpus not die that sounds
pretty good. But we still need a more generic solution for the long-term,
see below

 Is there really no way this can be handled closer to the source of
 the problem, in the i915 driver itself?  I do not know the flow of
 control in i915 (and X) at all, but on the surface it seems that the
 problem only comes when you map these problematic pages into the GTT
 (if I'm even using the right terminology), and something (not shmem.c)
 actively has to do that.
 
 Can't you check the pfn at that point, and if it's an unsuitable page,
 copy into a suitable page (perhaps allocated then, perhaps from a pool
 you primed earlier) and map that suitable page into the GTT instead?
 Maybe using page-private to link them if that helps.
 
 So long as the page (or its shadow) is mapped into the GTT, I imagine
 it would be pinned, and not liable to be swapped out or otherwise
 interfered with by shmem.c.  And when you unmap it from GTT, copy
 back to the unsuitable shmem object page before unpinning.
 
 I fully accept that I have very little understanding of GPU DRM GTT
 and i915, and this may be impossible or incoherent: but please, let's
 try to keep the strangeness where it belongs.  If necessary, we'll
 have add some kind of flag and callback from shmem.c to the driver;
 but I'd so much prefer to avoid that.

The copy stuff backforth approach is pretty much what ttm uses atm: It
allocates suitable pages with whatever means it has (usually through the
dma api) and if the shrinker callback tells it that it's sitting on too
much memory, it copies stuff out to the shmem backing storage used by gem.

There are quite a few issues with that approach:
- We expose mmap to the shmem file directly to userspace in i915. We use
  these extensively on Sandybridge because there direct cpu access is
  coherent with what the gpu does. Original userspace would always tell
  the kernel when it's done writing through cpu mappings so that the
  

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 08:25:38AM -0700, Linus Torvalds wrote:
> On Mon, May 7, 2012 at 4:13 PM, St?phane Marchesin  
> wrote:
> >
> > In the end, I came up with the ugly workaround of just leaking the
> > offending pages in shmem.c.
> 
> Don't leak it.
> 
> Instead, add it to some RCU list, and free it using RCU. Or some
> one-second timer or something.
> 
> That kind of approach should guarantee that it
> 
>  (a) gets returned to the system
> 
> but
> 
>  (b) the returning to the system gets delayed sufficiently that if the
> i915 driver is doing lots of allocations it will be getting other
> pages.
> 
> Hmm?

The problem is also that this only affects Sandybdrige gpus, so we'd need
to funnel this down to shmfs somehow ... Rob Clarke from Linaro will be
working on a gemfs to make backing storage allocation more flexible - they
need that to support some arm gpus. That way round we wouldn't need to put
some ugly drm/i915 stuff into core shmfs. Rob?
-Daniel
-- 
Daniel Vetter
Mail: daniel at ffwll.ch
Mobile: +41 (0)79 365 57 48

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Hugh Dickins]

On Mon, 7 May 2012, Stephane Marchesin wrote:

> While investing some Sandy Bridge rendering corruption, I found out
> that all physical memory pages below 1MiB were returning garbage when
> read through the GTT. This has been causing graphics corruption (when
> it's used for textures, render targets and pixmaps) and GPU hangups
> (when it's used for GPU batch buffers).
> 
> I talked with some people at Intel and they confirmed my findings,
> and said that a couple of other random pages were also affected.
> 
> We could fix this problem by adding an e820 region preventing the
> memory below 1 MiB to be used, but that prevents at least my machine
> from booting. One could think that we should be able to fix it in
> i915, but since the allocation is done by the backing shmem this is
> not possible.
> 
> In the end, I came up with the ugly workaround of just leaking the
> offending pages in shmem.c. I do realize it's truly ugly, but I'm
> looking for a fix to the existing code, and am wondering if people on
> this list have a better idea, short of rewriting i915_gem.c to
> allocate its own pages directly.
> 
> Signed-off-by: Stephane Marchesin 

Well done for discovering and pursuing this issue, but of course (as
you know: you're trying to provoke us to better) your patch is revolting.

And not even enough: swapin readahead and swapoff can read back
from swap into pages which the i915 will later turn out to dislike.

I do have a shmem.c patch coming up for gma500, which cannot use pages
over 4GB; but that fits more reasonably with memory allocation policies,
where we expect that anyone who can use a high page can use a lower as
well, and there's already __GFP_DMA32 to set the limit.

Your limitation is at the opposite end, so that patch won't help you at
all.  And I don't see how Andi's ZONE_DMA exclusion would work, without
equal hackery to enable private zonelists, avoiding that convention.

i915 is not the only user of shmem, and x86 not the only architecture:
we're not going to make everyone suffer for this.  Once the memory
allocator gets down to giving you the low 1MB, my guess is that it's
already short of memory, and liable to deadlock or OOM if you refuse
and soak up every page it then gives you.  Even if i915 has to live
with that possibility, we're not going to extend it to everyone else.

arch/x86/Kconfig has X86_RESERVE_LOW, default 64, range 4 640 (and
I think we reserve all the memory range from 640kB to 1MB anyway).
Would setting that to 640 allow you to boot, and avoid the i915
problem on all but the odd five pages?  I'm not pretending that's
an ideal solution at all (unless freeing initmem could release most
of it on non-SandyBridge and non-i915 machines), but it would be
useful to know if that does provide a stopgap solution.  If that
does work, maybe we just mark the odd five PageReserved at startup.

Is there really no way this can be handled closer to the source of
the problem, in the i915 driver itself?  I do not know the flow of
control in i915 (and X) at all, but on the surface it seems that the
problem only comes when you map these problematic pages into the GTT
(if I'm even using the right terminology), and something (not shmem.c)
actively has to do that.

Can't you check the pfn at that point, and if it's an unsuitable page,
copy into a suitable page (perhaps allocated then, perhaps from a pool
you primed earlier) and map that suitable page into the GTT instead?
Maybe using page->private to link them if that helps.

So long as the page (or its shadow) is mapped into the GTT, I imagine
it would be pinned, and not liable to be swapped out or otherwise
interfered with by shmem.c.  And when you unmap it from GTT, copy
back to the unsuitable shmem object page before unpinning.

I fully accept that I have very little understanding of GPU DRM GTT
and i915, and this may be impossible or incoherent: but please, let's
try to keep the strangeness where it belongs.  If necessary, we'll
have add some kind of flag and callback from shmem.c to the driver;
but I'd so much prefer to avoid that.

Hugh

> 
> Change-Id: I957e125fb280e0b0d6b05a83cc4068df2f05aa0a
> ---
>  mm/shmem.c |   39 +--
>  1 files changed, 37 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 6c253f7..dcbb58b 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -768,6 +768,31 @@ redirty:
>   return 0;
>  }
>  
> +/*
> + * Some intel GPUs can't use those pages in the GTT, which results in
> + * graphics corruption. Sadly, it's impossible to prevent usage of those
> + * pages in the intel allocator.
> + *
> + * Instead, we test for those areas here and leak the corresponding pages.
> + *
> + * Some day, when the intel GPU memory is not backed by shmem any more,
> + * we'll be able to come up with a solution which is contained in i915.
> + */
> +static bool i915_usable_page(struct page *page)
> +{
> + dma_addr_t addr = page_to_phys(page);
> +
> + if 

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Rob Clark]

On Tue, May 8, 2012 at 12:23 PM, Daniel Vetter  wrote:
> On Tue, May 08, 2012 at 08:25:38AM -0700, Linus Torvalds wrote:
>> On Mon, May 7, 2012 at 4:13 PM, St?phane Marchesin  
>> wrote:
>> >
>> > In the end, I came up with the ugly workaround of just leaking the
>> > offending pages in shmem.c.
>>
>> Don't leak it.
>>
>> Instead, add it to some RCU list, and free it using RCU. Or some
>> one-second timer or something.
>>
>> That kind of approach should guarantee that it
>>
>> ?(a) gets returned to the system
>>
>> but
>>
>> ?(b) the returning to the system gets delayed sufficiently that if the
>> i915 driver is doing lots of allocations it will be getting other
>> pages.
>>
>> Hmm?
>
> The problem is also that this only affects Sandybdrige gpus, so we'd need
> to funnel this down to shmfs somehow ... Rob Clarke from Linaro will be
> working on a gemfs to make backing storage allocation more flexible - they
> need that to support some arm gpus. That way round we wouldn't need to put
> some ugly drm/i915 stuff into core shmfs. Rob?

Well, a bit hand-wavey at this point, but the idea is to let the
driver have control of the page allocation via 'struct
address_space_operations'.. but otherwise work in a similar way as
shmfs.

Something like get_xip_mem() is almost what we want, except we don't
want it to populate the pages, we don't want to force a kernel
mapping, and shmem doesn't use it..

I suppose we still need a short term fix for i915, but at least it
would only be temporary.

BR,
-R

> -Daniel
> --
> Daniel Vetter
> Mail: daniel at ffwll.ch
> Mobile: +41 (0)79 365 57 48
> ___
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 07:53:36AM +0100, Dave Airlie wrote:
> On Tue, May 8, 2012 at 12:13 AM, St?phane Marchesin
>  wrote:
> > While investing some Sandy Bridge rendering corruption, I found out
> > that all physical memory pages below 1MiB were returning garbage when
> > read through the GTT. This has been causing graphics corruption (when
> > it's used for textures, render targets and pixmaps) and GPU hangups
> > (when it's used for GPU batch buffers).
> >
> > I talked with some people at Intel and they confirmed my findings,
> > and said that a couple of other random pages were also affected.
> >
> > We could fix this problem by adding an e820 region preventing the
> > memory below 1 MiB to be used, but that prevents at least my machine
> > from booting. One could think that we should be able to fix it in
> > i915, but since the allocation is done by the backing shmem this is
> > not possible.
> >
> > In the end, I came up with the ugly workaround of just leaking the
> > offending pages in shmem.c. I do realize it's truly ugly, but I'm
> > looking for a fix to the existing code, and am wondering if people on
> > this list have a better idea, short of rewriting i915_gem.c to
> > allocate its own pages directly.
> 
> Ouch, can Intel get some details on why these pages are "special" and
> if they are special across chipsets, Ironlake? Ivybridge?
> 
> Like I can handle the < 1MB but the other selected pages look pretty
> random or misc, (2005, 2011, 2013? years?, 40004000, some shout out to
> the 4004?

I've discussed this with St?phane, and that enumeration of 5 pages is
exhaustive. And both these 4 pages and the low 1mb block only cause
problems on snb (ivb and later is fixed). For the special pages the
official workaround is that the bios marks the two 2M blocks of memory at
512M and 1024M as reserved. And for the low 1M I guess Windows doesn't
hand out any of these to device drivers.
-Daniel
-- 
Daniel Vetter
Mail: daniel at ffwll.ch
Mobile: +41 (0)79 365 57 48

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Linus Torvalds]

On Mon, May 7, 2012 at 4:13 PM, St?phane Marchesin  
wrote:
>
> In the end, I came up with the ugly workaround of just leaking the
> offending pages in shmem.c.

Don't leak it.

Instead, add it to some RCU list, and free it using RCU. Or some
one-second timer or something.

That kind of approach should guarantee that it

 (a) gets returned to the system

but

 (b) the returning to the system gets delayed sufficiently that if the
i915 driver is doing lots of allocations it will be getting other
pages.

Hmm?

  Linus

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Dave Airlie]

On Tue, May 8, 2012 at 12:13 AM, St?phane Marchesin
 wrote:
> While investing some Sandy Bridge rendering corruption, I found out
> that all physical memory pages below 1MiB were returning garbage when
> read through the GTT. This has been causing graphics corruption (when
> it's used for textures, render targets and pixmaps) and GPU hangups
> (when it's used for GPU batch buffers).
>
> I talked with some people at Intel and they confirmed my findings,
> and said that a couple of other random pages were also affected.
>
> We could fix this problem by adding an e820 region preventing the
> memory below 1 MiB to be used, but that prevents at least my machine
> from booting. One could think that we should be able to fix it in
> i915, but since the allocation is done by the backing shmem this is
> not possible.
>
> In the end, I came up with the ugly workaround of just leaking the
> offending pages in shmem.c. I do realize it's truly ugly, but I'm
> looking for a fix to the existing code, and am wondering if people on
> this list have a better idea, short of rewriting i915_gem.c to
> allocate its own pages directly.

Ouch, can Intel get some details on why these pages are "special" and
if they are special across chipsets, Ironlake? Ivybridge?

Like I can handle the < 1MB but the other selected pages look pretty
random or misc, (2005, 2011, 2013? years?, 40004000, some shout out to
the 4004?

Dave.

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Stéphane Marchesin]

While investing some Sandy Bridge rendering corruption, I found out
that all physical memory pages below 1MiB were returning garbage when
read through the GTT. This has been causing graphics corruption (when
it's used for textures, render targets and pixmaps) and GPU hangups
(when it's used for GPU batch buffers).

I talked with some people at Intel and they confirmed my findings,
and said that a couple of other random pages were also affected.

We could fix this problem by adding an e820 region preventing the
memory below 1 MiB to be used, but that prevents at least my machine
from booting. One could think that we should be able to fix it in
i915, but since the allocation is done by the backing shmem this is
not possible.

In the end, I came up with the ugly workaround of just leaking the
offending pages in shmem.c. I do realize it's truly ugly, but I'm
looking for a fix to the existing code, and am wondering if people on
this list have a better idea, short of rewriting i915_gem.c to
allocate its own pages directly.

Signed-off-by: Stéphane Marchesin marc...@chromium.org

Change-Id: I957e125fb280e0b0d6b05a83cc4068df2f05aa0a
---
 mm/shmem.c |   39 +--
 1 files changed, 37 insertions(+), 2 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 6c253f7..dcbb58b 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -768,6 +768,31 @@ redirty:
return 0;
 }
 
+/*
+ * Some intel GPUs can't use those pages in the GTT, which results in
+ * graphics corruption. Sadly, it's impossible to prevent usage of those
+ * pages in the intel allocator.
+ *
+ * Instead, we test for those areas here and leak the corresponding pages.
+ *
+ * Some day, when the intel GPU memory is not backed by shmem any more,
+ * we'll be able to come up with a solution which is contained in i915.
+ */
+static bool i915_usable_page(struct page *page)
+{
+   dma_addr_t addr = page_to_phys(page);
+
+   if (unlikely((addr  1 * 1024 * 1024) ||
+   (addr == 0x2005) ||
+   (addr == 0x2011) ||
+   (addr == 0x2013) ||
+   (addr == 0x20138000) ||
+   (addr == 0x40004000)))
+   return false;
+
+   return true;
+}
+
 #ifdef CONFIG_NUMA
 #ifdef CONFIG_TMPFS
 static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
@@ -816,6 +841,7 @@ static struct page *shmem_alloc_page(gfp_t gfp,
struct shmem_inode_info *info, pgoff_t index)
 {
struct vm_area_struct pvma;
+   struct page *page;
 
/* Create a pseudo vma that just contains the policy */
pvma.vm_start = 0;
@@ -826,7 +852,11 @@ static struct page *shmem_alloc_page(gfp_t gfp,
/*
 * alloc_page_vma() will drop the shared policy reference
 */
-   return alloc_page_vma(gfp, pvma, 0);
+   do {
+   page = alloc_page_vma(gfp, pvma, 0);
+   } while (!i915_usable_page(page));
+
+   return page;
 }
 #else /* !CONFIG_NUMA */
 #ifdef CONFIG_TMPFS
@@ -844,7 +874,12 @@ static inline struct page *shmem_swapin(swp_entry_t swap, 
gfp_t gfp,
 static inline struct page *shmem_alloc_page(gfp_t gfp,
struct shmem_inode_info *info, pgoff_t index)
 {
-   return alloc_page(gfp);
+   struct page *page;
+   do {
+   page = alloc_page(gfp);
+   } while (!i915_usable_page(page));
+
+   return page;
 }
 #endif /* CONFIG_NUMA */
 
-- 
1.7.5.3.367.ga9930

___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Andi Kleen]

Stéphane Marchesin marc...@chromium.org writes:

 While investing some Sandy Bridge rendering corruption, I found out
 that all physical memory pages below 1MiB were returning garbage when
 read through the GTT. This has been causing graphics corruption (when
 it's used for textures, render targets and pixmaps) and GPU hangups
 (when it's used for GPU batch buffers).

It would be possible to exlude GFP_DMA from the page allocator. That
covers the first 16MB. You just need a custom zone list with ZONE_DMA.

-Andi

-- 
a...@linux.intel.com -- Speaking for myself only
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Dave Airlie]

On Tue, May 8, 2012 at 12:13 AM, Stéphane Marchesin
marc...@chromium.org wrote:
 While investing some Sandy Bridge rendering corruption, I found out
 that all physical memory pages below 1MiB were returning garbage when
 read through the GTT. This has been causing graphics corruption (when
 it's used for textures, render targets and pixmaps) and GPU hangups
 (when it's used for GPU batch buffers).

 I talked with some people at Intel and they confirmed my findings,
 and said that a couple of other random pages were also affected.

 We could fix this problem by adding an e820 region preventing the
 memory below 1 MiB to be used, but that prevents at least my machine
 from booting. One could think that we should be able to fix it in
 i915, but since the allocation is done by the backing shmem this is
 not possible.

 In the end, I came up with the ugly workaround of just leaking the
 offending pages in shmem.c. I do realize it's truly ugly, but I'm
 looking for a fix to the existing code, and am wondering if people on
 this list have a better idea, short of rewriting i915_gem.c to
 allocate its own pages directly.

Ouch, can Intel get some details on why these pages are special and
if they are special across chipsets, Ironlake? Ivybridge?

Like I can handle the  1MB but the other selected pages look pretty
random or misc, (2005, 2011, 2013? years?, 40004000, some shout out to
the 4004?

Dave.
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 07:53:36AM +0100, Dave Airlie wrote:
 On Tue, May 8, 2012 at 12:13 AM, Stéphane Marchesin
 marc...@chromium.org wrote:
  While investing some Sandy Bridge rendering corruption, I found out
  that all physical memory pages below 1MiB were returning garbage when
  read through the GTT. This has been causing graphics corruption (when
  it's used for textures, render targets and pixmaps) and GPU hangups
  (when it's used for GPU batch buffers).
 
  I talked with some people at Intel and they confirmed my findings,
  and said that a couple of other random pages were also affected.
 
  We could fix this problem by adding an e820 region preventing the
  memory below 1 MiB to be used, but that prevents at least my machine
  from booting. One could think that we should be able to fix it in
  i915, but since the allocation is done by the backing shmem this is
  not possible.
 
  In the end, I came up with the ugly workaround of just leaking the
  offending pages in shmem.c. I do realize it's truly ugly, but I'm
  looking for a fix to the existing code, and am wondering if people on
  this list have a better idea, short of rewriting i915_gem.c to
  allocate its own pages directly.
 
 Ouch, can Intel get some details on why these pages are special and
 if they are special across chipsets, Ironlake? Ivybridge?
 
 Like I can handle the  1MB but the other selected pages look pretty
 random or misc, (2005, 2011, 2013? years?, 40004000, some shout out to
 the 4004?

I've discussed this with Stéphane, and that enumeration of 5 pages is
exhaustive. And both these 4 pages and the low 1mb block only cause
problems on snb (ivb and later is fixed). For the special pages the
official workaround is that the bios marks the two 2M blocks of memory at
512M and 1024M as reserved. And for the low 1M I guess Windows doesn't
hand out any of these to device drivers.
-Daniel
-- 
Daniel Vetter
Mail: dan...@ffwll.ch
Mobile: +41 (0)79 365 57 48
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Daniel Vetter]

On Tue, May 08, 2012 at 08:25:38AM -0700, Linus Torvalds wrote:
 On Mon, May 7, 2012 at 4:13 PM, Stéphane Marchesin marc...@chromium.org 
 wrote:
 
  In the end, I came up with the ugly workaround of just leaking the
  offending pages in shmem.c.
 
 Don't leak it.
 
 Instead, add it to some RCU list, and free it using RCU. Or some
 one-second timer or something.
 
 That kind of approach should guarantee that it
 
  (a) gets returned to the system
 
 but
 
  (b) the returning to the system gets delayed sufficiently that if the
 i915 driver is doing lots of allocations it will be getting other
 pages.
 
 Hmm?

The problem is also that this only affects Sandybdrige gpus, so we'd need
to funnel this down to shmfs somehow ... Rob Clarke from Linaro will be
working on a gemfs to make backing storage allocation more flexible - they
need that to support some arm gpus. That way round we wouldn't need to put
some ugly drm/i915 stuff into core shmfs. Rob?
-Daniel
-- 
Daniel Vetter
Mail: dan...@ffwll.ch
Mobile: +41 (0)79 365 57 48
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Rob Clark]

On Tue, May 8, 2012 at 12:23 PM, Daniel Vetter dan...@ffwll.ch wrote:
 On Tue, May 08, 2012 at 08:25:38AM -0700, Linus Torvalds wrote:
 On Mon, May 7, 2012 at 4:13 PM, Stéphane Marchesin marc...@chromium.org 
 wrote:
 
  In the end, I came up with the ugly workaround of just leaking the
  offending pages in shmem.c.

 Don't leak it.

 Instead, add it to some RCU list, and free it using RCU. Or some
 one-second timer or something.

 That kind of approach should guarantee that it

  (a) gets returned to the system

 but

  (b) the returning to the system gets delayed sufficiently that if the
 i915 driver is doing lots of allocations it will be getting other
 pages.

 Hmm?

 The problem is also that this only affects Sandybdrige gpus, so we'd need
 to funnel this down to shmfs somehow ... Rob Clarke from Linaro will be
 working on a gemfs to make backing storage allocation more flexible - they
 need that to support some arm gpus. That way round we wouldn't need to put
 some ugly drm/i915 stuff into core shmfs. Rob?

Well, a bit hand-wavey at this point, but the idea is to let the
driver have control of the page allocation via 'struct
address_space_operations'.. but otherwise work in a similar way as
shmfs.

Something like get_xip_mem() is almost what we want, except we don't
want it to populate the pages, we don't want to force a kernel
mapping, and shmem doesn't use it..

I suppose we still need a short term fix for i915, but at least it
would only be temporary.

BR,
-R

 -Daniel
 --
 Daniel Vetter
 Mail: dan...@ffwll.ch
 Mobile: +41 (0)79 365 57 48
 ___
 dri-devel mailing list
 dri-devel@lists.freedesktop.org
 http://lists.freedesktop.org/mailman/listinfo/dri-devel
___
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Andi Kleen]

St?phane Marchesin  writes:

> While investing some Sandy Bridge rendering corruption, I found out
> that all physical memory pages below 1MiB were returning garbage when
> read through the GTT. This has been causing graphics corruption (when
> it's used for textures, render targets and pixmaps) and GPU hangups
> (when it's used for GPU batch buffers).

It would be possible to exlude GFP_DMA from the page allocator. That
covers the first 16MB. You just need a custom zone list with ZONE_DMA.

-Andi

-- 
ak at linux.intel.com -- Speaking for myself only

[PATCH] mm: Work around Intel SNB GTT bug with some physical pages.

[Stéphane Marchesin]

While investing some Sandy Bridge rendering corruption, I found out
that all physical memory pages below 1MiB were returning garbage when
read through the GTT. This has been causing graphics corruption (when
it's used for textures, render targets and pixmaps) and GPU hangups
(when it's used for GPU batch buffers).

I talked with some people at Intel and they confirmed my findings,
and said that a couple of other random pages were also affected.

We could fix this problem by adding an e820 region preventing the
memory below 1 MiB to be used, but that prevents at least my machine
from booting. One could think that we should be able to fix it in
i915, but since the allocation is done by the backing shmem this is
not possible.

In the end, I came up with the ugly workaround of just leaking the
offending pages in shmem.c. I do realize it's truly ugly, but I'm
looking for a fix to the existing code, and am wondering if people on
this list have a better idea, short of rewriting i915_gem.c to
allocate its own pages directly.

Signed-off-by: St?phane Marchesin 

Change-Id: I957e125fb280e0b0d6b05a83cc4068df2f05aa0a
---
 mm/shmem.c |   39 +--
 1 files changed, 37 insertions(+), 2 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 6c253f7..dcbb58b 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -768,6 +768,31 @@ redirty:
return 0;
 }

+/*
+ * Some intel GPUs can't use those pages in the GTT, which results in
+ * graphics corruption. Sadly, it's impossible to prevent usage of those
+ * pages in the intel allocator.
+ *
+ * Instead, we test for those areas here and leak the corresponding pages.
+ *
+ * Some day, when the intel GPU memory is not backed by shmem any more,
+ * we'll be able to come up with a solution which is contained in i915.
+ */
+static bool i915_usable_page(struct page *page)
+{
+   dma_addr_t addr = page_to_phys(page);
+
+   if (unlikely((addr < 1 * 1024 * 1024) ||
+   (addr == 0x2005) ||
+   (addr == 0x2011) ||
+   (addr == 0x2013) ||
+   (addr == 0x20138000) ||
+   (addr == 0x40004000)))
+   return false;
+
+   return true;
+}
+
 #ifdef CONFIG_NUMA
 #ifdef CONFIG_TMPFS
 static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
@@ -816,6 +841,7 @@ static struct page *shmem_alloc_page(gfp_t gfp,
struct shmem_inode_info *info, pgoff_t index)
 {
struct vm_area_struct pvma;
+   struct page *page;

/* Create a pseudo vma that just contains the policy */
pvma.vm_start = 0;
@@ -826,7 +852,11 @@ static struct page *shmem_alloc_page(gfp_t gfp,
/*
 * alloc_page_vma() will drop the shared policy reference
 */
-   return alloc_page_vma(gfp, , 0);
+   do {
+   page = alloc_page_vma(gfp, , 0);
+   } while (!i915_usable_page(page));
+
+   return page;
 }
 #else /* !CONFIG_NUMA */
 #ifdef CONFIG_TMPFS
@@ -844,7 +874,12 @@ static inline struct page *shmem_swapin(swp_entry_t swap, 
gfp_t gfp,
 static inline struct page *shmem_alloc_page(gfp_t gfp,
struct shmem_inode_info *info, pgoff_t index)
 {
-   return alloc_page(gfp);
+   struct page *page;
+   do {
+   page = alloc_page(gfp);
+   } while (!i915_usable_page(page));
+
+   return page;
 }
 #endif /* CONFIG_NUMA */

-- 
1.7.5.3.367.ga9930