Re: [PATCH 00/18] introduce a new tool, valid access checker

[Andrew Morton]

On Fri, 22 Dec 2017 10:51:15 +0900 Joonsoo Kim  wrote:

> On Tue, Nov 28, 2017 at 04:48:35PM +0900, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> > 
> > Hello,
> > 
> > This patchset introduces a new tool, valid access checker.
> > 
> > Vchecker is a dynamic memory error detector. It provides a new debug feature
> > that can find out an un-intended access to valid area. Valid area here means
> > the memory which is allocated and allowed to be accessed by memory owner and
> > un-intended access means the read/write that is initiated by non-owner.
> > Usual problem of this class is memory overwritten.
> > 
> > Most of debug feature focused on finding out un-intended access to
> > in-valid area, for example, out-of-bound access and use-after-free, and,
> > there are many good tools for it. But, as far as I know, there is no good 
> > tool
> > to find out un-intended access to valid area. This kind of problem is really
> > hard to solve so this tool would be very useful.
> > 
> > This tool doesn't automatically catch a problem. Manual runtime 
> > configuration
> > to specify the target object is required.
> > 
> > Note that there was a similar attempt for the debugging overwritten problem
> > however it requires manual code modifying and recompile.
> > 
> > http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> > 
> > To get more information about vchecker, please see a documention at
> > the last patch.
> > 
> > Patchset can also be available at
> > 
> > https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> > 
> > Enjoy it.
> > 
> > Thanks.
> 
> Hello, Andrew.
> 
> Before the fixing some build failure on this patchset, I'd like to know
> other reviewer's opinion on this patchset, especially, yours. :)
> 
> There are some interests on this patchset from some developers. Wengang
> come up with a very similar change and Andi said that this looks useful.
> Do you think that this tool is useful and can be merged?
> 

My main fear is that the feature will sit there and nobody will use it.

Are there ways around that?  For example, can we arrange with the test
robot(s) to get vchecker operating on their setups in some automatable
fashion and have them checking for bugs?

Any other suggestions as to how we could get this feature to be used by
others and producing useful results?

And has vchecker actually found any real bugs in existing code?  If so,
a description of that would be illuminating.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Andrew Morton]

On Fri, 22 Dec 2017 10:51:15 +0900 Joonsoo Kim  wrote:

> On Tue, Nov 28, 2017 at 04:48:35PM +0900, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> > 
> > Hello,
> > 
> > This patchset introduces a new tool, valid access checker.
> > 
> > Vchecker is a dynamic memory error detector. It provides a new debug feature
> > that can find out an un-intended access to valid area. Valid area here means
> > the memory which is allocated and allowed to be accessed by memory owner and
> > un-intended access means the read/write that is initiated by non-owner.
> > Usual problem of this class is memory overwritten.
> > 
> > Most of debug feature focused on finding out un-intended access to
> > in-valid area, for example, out-of-bound access and use-after-free, and,
> > there are many good tools for it. But, as far as I know, there is no good 
> > tool
> > to find out un-intended access to valid area. This kind of problem is really
> > hard to solve so this tool would be very useful.
> > 
> > This tool doesn't automatically catch a problem. Manual runtime 
> > configuration
> > to specify the target object is required.
> > 
> > Note that there was a similar attempt for the debugging overwritten problem
> > however it requires manual code modifying and recompile.
> > 
> > http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> > 
> > To get more information about vchecker, please see a documention at
> > the last patch.
> > 
> > Patchset can also be available at
> > 
> > https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> > 
> > Enjoy it.
> > 
> > Thanks.
> 
> Hello, Andrew.
> 
> Before the fixing some build failure on this patchset, I'd like to know
> other reviewer's opinion on this patchset, especially, yours. :)
> 
> There are some interests on this patchset from some developers. Wengang
> come up with a very similar change and Andi said that this looks useful.
> Do you think that this tool is useful and can be merged?
> 

My main fear is that the feature will sit there and nobody will use it.

Are there ways around that?  For example, can we arrange with the test
robot(s) to get vchecker operating on their setups in some automatable
fashion and have them checking for bugs?

Any other suggestions as to how we could get this feature to be used by
others and producing useful results?

And has vchecker actually found any real bugs in existing code?  If so,
a description of that would be illuminating.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Joonsoo Kim]

On Tue, Nov 28, 2017 at 04:48:35PM +0900, js1...@gmail.com wrote:
> From: Joonsoo Kim 
> 
> Hello,
> 
> This patchset introduces a new tool, valid access checker.
> 
> Vchecker is a dynamic memory error detector. It provides a new debug feature
> that can find out an un-intended access to valid area. Valid area here means
> the memory which is allocated and allowed to be accessed by memory owner and
> un-intended access means the read/write that is initiated by non-owner.
> Usual problem of this class is memory overwritten.
> 
> Most of debug feature focused on finding out un-intended access to
> in-valid area, for example, out-of-bound access and use-after-free, and,
> there are many good tools for it. But, as far as I know, there is no good tool
> to find out un-intended access to valid area. This kind of problem is really
> hard to solve so this tool would be very useful.
> 
> This tool doesn't automatically catch a problem. Manual runtime configuration
> to specify the target object is required.
> 
> Note that there was a similar attempt for the debugging overwritten problem
> however it requires manual code modifying and recompile.
> 
> http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> 
> To get more information about vchecker, please see a documention at
> the last patch.
> 
> Patchset can also be available at
> 
> https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> 
> Enjoy it.
> 
> Thanks.

Hello, Andrew.

Before the fixing some build failure on this patchset, I'd like to know
other reviewer's opinion on this patchset, especially, yours. :)

There are some interests on this patchset from some developers. Wengang
come up with a very similar change and Andi said that this looks useful.
Do you think that this tool is useful and can be merged?

Thanks.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Joonsoo Kim]

On Tue, Nov 28, 2017 at 04:48:35PM +0900, js1...@gmail.com wrote:
> From: Joonsoo Kim 
> 
> Hello,
> 
> This patchset introduces a new tool, valid access checker.
> 
> Vchecker is a dynamic memory error detector. It provides a new debug feature
> that can find out an un-intended access to valid area. Valid area here means
> the memory which is allocated and allowed to be accessed by memory owner and
> un-intended access means the read/write that is initiated by non-owner.
> Usual problem of this class is memory overwritten.
> 
> Most of debug feature focused on finding out un-intended access to
> in-valid area, for example, out-of-bound access and use-after-free, and,
> there are many good tools for it. But, as far as I know, there is no good tool
> to find out un-intended access to valid area. This kind of problem is really
> hard to solve so this tool would be very useful.
> 
> This tool doesn't automatically catch a problem. Manual runtime configuration
> to specify the target object is required.
> 
> Note that there was a similar attempt for the debugging overwritten problem
> however it requires manual code modifying and recompile.
> 
> http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> 
> To get more information about vchecker, please see a documention at
> the last patch.
> 
> Patchset can also be available at
> 
> https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> 
> Enjoy it.
> 
> Thanks.

Hello, Andrew.

Before the fixing some build failure on this patchset, I'd like to know
other reviewer's opinion on this patchset, especially, yours. :)

There are some interests on this patchset from some developers. Wengang
come up with a very similar change and Andi said that this looks useful.
Do you think that this tool is useful and can be merged?

Thanks.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Joonsoo Kim]

On Wed, Nov 29, 2017 at 10:27:00AM +0100, Dmitry Vyukov wrote:
> On Tue, Nov 28, 2017 at 8:48 AM,   wrote:
> > From: Joonsoo Kim 
> >
> > Hello,
> >
> > This patchset introduces a new tool, valid access checker.
> >
> > Vchecker is a dynamic memory error detector. It provides a new debug feature
> > that can find out an un-intended access to valid area. Valid area here means
> > the memory which is allocated and allowed to be accessed by memory owner and
> > un-intended access means the read/write that is initiated by non-owner.
> > Usual problem of this class is memory overwritten.
> >
> > Most of debug feature focused on finding out un-intended access to
> > in-valid area, for example, out-of-bound access and use-after-free, and,
> > there are many good tools for it. But, as far as I know, there is no good 
> > tool
> > to find out un-intended access to valid area. This kind of problem is really
> > hard to solve so this tool would be very useful.
> >
> > This tool doesn't automatically catch a problem. Manual runtime 
> > configuration
> > to specify the target object is required.
> >
> > Note that there was a similar attempt for the debugging overwritten problem
> > however it requires manual code modifying and recompile.
> >
> > http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> >
> > To get more information about vchecker, please see a documention at
> > the last patch.
> >
> > Patchset can also be available at
> >
> > https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> >
> > Enjoy it.
> 
> 
> Hi Joonsoo,
> 
> I skimmed through the code and this looks fine from KASAN point of
> view (minimal code changes and no perf impact).
> I don't feel like I can judge if this should go in or not. I will not
> use this, we use KASAN for large-scale testing, but vchecker is in a
> different bucket, it is meant for developers debugging hard bugs.
> Wengang come up with a very similar change, and Andi said that this
> looks useful.

Thanks for comment.

Hello, other reviewers!
Please let me know more opinions about this patchset.

> 
> If the decision is that this goes in, please let me take a closer look
> before this is merged.

I will let you know when the decision is made.

Thanks.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Joonsoo Kim]

On Wed, Nov 29, 2017 at 10:27:00AM +0100, Dmitry Vyukov wrote:
> On Tue, Nov 28, 2017 at 8:48 AM,   wrote:
> > From: Joonsoo Kim 
> >
> > Hello,
> >
> > This patchset introduces a new tool, valid access checker.
> >
> > Vchecker is a dynamic memory error detector. It provides a new debug feature
> > that can find out an un-intended access to valid area. Valid area here means
> > the memory which is allocated and allowed to be accessed by memory owner and
> > un-intended access means the read/write that is initiated by non-owner.
> > Usual problem of this class is memory overwritten.
> >
> > Most of debug feature focused on finding out un-intended access to
> > in-valid area, for example, out-of-bound access and use-after-free, and,
> > there are many good tools for it. But, as far as I know, there is no good 
> > tool
> > to find out un-intended access to valid area. This kind of problem is really
> > hard to solve so this tool would be very useful.
> >
> > This tool doesn't automatically catch a problem. Manual runtime 
> > configuration
> > to specify the target object is required.
> >
> > Note that there was a similar attempt for the debugging overwritten problem
> > however it requires manual code modifying and recompile.
> >
> > http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
> >
> > To get more information about vchecker, please see a documention at
> > the last patch.
> >
> > Patchset can also be available at
> >
> > https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
> >
> > Enjoy it.
> 
> 
> Hi Joonsoo,
> 
> I skimmed through the code and this looks fine from KASAN point of
> view (minimal code changes and no perf impact).
> I don't feel like I can judge if this should go in or not. I will not
> use this, we use KASAN for large-scale testing, but vchecker is in a
> different bucket, it is meant for developers debugging hard bugs.
> Wengang come up with a very similar change, and Andi said that this
> looks useful.

Thanks for comment.

Hello, other reviewers!
Please let me know more opinions about this patchset.

> 
> If the decision is that this goes in, please let me take a closer look
> before this is merged.

I will let you know when the decision is made.

Thanks.

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Dmitry Vyukov]

On Tue, Nov 28, 2017 at 8:48 AM,   wrote:
> From: Joonsoo Kim 
>
> Hello,
>
> This patchset introduces a new tool, valid access checker.
>
> Vchecker is a dynamic memory error detector. It provides a new debug feature
> that can find out an un-intended access to valid area. Valid area here means
> the memory which is allocated and allowed to be accessed by memory owner and
> un-intended access means the read/write that is initiated by non-owner.
> Usual problem of this class is memory overwritten.
>
> Most of debug feature focused on finding out un-intended access to
> in-valid area, for example, out-of-bound access and use-after-free, and,
> there are many good tools for it. But, as far as I know, there is no good tool
> to find out un-intended access to valid area. This kind of problem is really
> hard to solve so this tool would be very useful.
>
> This tool doesn't automatically catch a problem. Manual runtime configuration
> to specify the target object is required.
>
> Note that there was a similar attempt for the debugging overwritten problem
> however it requires manual code modifying and recompile.
>
> http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
>
> To get more information about vchecker, please see a documention at
> the last patch.
>
> Patchset can also be available at
>
> https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
>
> Enjoy it.


Hi Joonsoo,

I skimmed through the code and this looks fine from KASAN point of
view (minimal code changes and no perf impact).
I don't feel like I can judge if this should go in or not. I will not
use this, we use KASAN for large-scale testing, but vchecker is in a
different bucket, it is meant for developers debugging hard bugs.
Wengang come up with a very similar change, and Andi said that this
looks useful.

If the decision is that this goes in, please let me take a closer look
before this is merged.

Thanks

Re: [PATCH 00/18] introduce a new tool, valid access checker

[Dmitry Vyukov]

On Tue, Nov 28, 2017 at 8:48 AM,   wrote:
> From: Joonsoo Kim 
>
> Hello,
>
> This patchset introduces a new tool, valid access checker.
>
> Vchecker is a dynamic memory error detector. It provides a new debug feature
> that can find out an un-intended access to valid area. Valid area here means
> the memory which is allocated and allowed to be accessed by memory owner and
> un-intended access means the read/write that is initiated by non-owner.
> Usual problem of this class is memory overwritten.
>
> Most of debug feature focused on finding out un-intended access to
> in-valid area, for example, out-of-bound access and use-after-free, and,
> there are many good tools for it. But, as far as I know, there is no good tool
> to find out un-intended access to valid area. This kind of problem is really
> hard to solve so this tool would be very useful.
>
> This tool doesn't automatically catch a problem. Manual runtime configuration
> to specify the target object is required.
>
> Note that there was a similar attempt for the debugging overwritten problem
> however it requires manual code modifying and recompile.
>
> http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>
>
> To get more information about vchecker, please see a documention at
> the last patch.
>
> Patchset can also be available at
>
> https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122
>
> Enjoy it.


Hi Joonsoo,

I skimmed through the code and this looks fine from KASAN point of
view (minimal code changes and no perf impact).
I don't feel like I can judge if this should go in or not. I will not
use this, we use KASAN for large-scale testing, but vchecker is in a
different bucket, it is meant for developers debugging hard bugs.
Wengang come up with a very similar change, and Andi said that this
looks useful.

If the decision is that this goes in, please let me take a closer look
before this is merged.

Thanks

[PATCH 00/18] introduce a new tool, valid access checker

[js1304]

From: Joonsoo Kim 

Hello,

This patchset introduces a new tool, valid access checker.

Vchecker is a dynamic memory error detector. It provides a new debug feature
that can find out an un-intended access to valid area. Valid area here means
the memory which is allocated and allowed to be accessed by memory owner and
un-intended access means the read/write that is initiated by non-owner.
Usual problem of this class is memory overwritten.

Most of debug feature focused on finding out un-intended access to
in-valid area, for example, out-of-bound access and use-after-free, and,
there are many good tools for it. But, as far as I know, there is no good tool
to find out un-intended access to valid area. This kind of problem is really
hard to solve so this tool would be very useful.

This tool doesn't automatically catch a problem. Manual runtime configuration
to specify the target object is required.

Note that there was a similar attempt for the debugging overwritten problem
however it requires manual code modifying and recompile.

http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>

To get more information about vchecker, please see a documention at
the last patch.

Patchset can also be available at

https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122

Enjoy it.

Thanks.

Joonsoo Kim (14):
  mm/kasan: make some kasan functions global
  vchecker: introduce the valid access checker
  vchecker: mark/unmark the shadow of the allocated objects
  vchecker: prepare per object memory for vchecker
  vchecker: store/report callstack of value writer
  lib/stackdepot: extend stackdepot API to support per-user stackdepot
  vchecker: consistently exclude vchecker's stacktrace
  vchecker: fix 'remove' handling on callstack checker
  mm/vchecker: support inline KASAN build
  mm/vchecker: make callstack depth configurable
  mm/vchecker: pass allocation caller address to vchecker hook
  mm/vchecker: support allocation caller filter
  lib/vchecker_test: introduce a sample for vchecker test
  doc: add vchecker document

Namhyung Kim (4):
  lib/stackdepot: Add is_new arg to depot_save_stack
  vchecker: Add 'callstack' checker
  vchecker: Support toggle on/off of callstack check
  vchecker: Use __GFP_ATOMIC to save stacktrace

 Documentation/dev-tools/vchecker.rst |  200 +++
 drivers/gpu/drm/drm_mm.c |4 +-
 include/linux/kasan.h|1 +
 include/linux/slab.h |8 +
 include/linux/slab_def.h |3 +
 include/linux/slub_def.h |3 +
 include/linux/stackdepot.h   |   10 +-
 lib/Kconfig.kasan|   21 +
 lib/Makefile |1 +
 lib/stackdepot.c |  126 ++--
 lib/vchecker_test.c  |  117 
 mm/kasan/Makefile|1 +
 mm/kasan/kasan.c |   14 +-
 mm/kasan/kasan.h |3 +
 mm/kasan/report.c|   12 +-
 mm/kasan/vchecker.c  | 1089 ++
 mm/kasan/vchecker.h  |   43 ++
 mm/page_owner.c  |8 +-
 mm/slab.c|   47 +-
 mm/slab.h|   14 +-
 mm/slab_common.c |   25 +
 mm/slub.c|   49 +-
 22 files changed, 1730 insertions(+), 69 deletions(-)
 create mode 100644 Documentation/dev-tools/vchecker.rst
 create mode 100644 lib/vchecker_test.c
 create mode 100644 mm/kasan/vchecker.c
 create mode 100644 mm/kasan/vchecker.h

-- 
2.7.4

[PATCH 00/18] introduce a new tool, valid access checker

[js1304]

From: Joonsoo Kim 

Hello,

This patchset introduces a new tool, valid access checker.

Vchecker is a dynamic memory error detector. It provides a new debug feature
that can find out an un-intended access to valid area. Valid area here means
the memory which is allocated and allowed to be accessed by memory owner and
un-intended access means the read/write that is initiated by non-owner.
Usual problem of this class is memory overwritten.

Most of debug feature focused on finding out un-intended access to
in-valid area, for example, out-of-bound access and use-after-free, and,
there are many good tools for it. But, as far as I know, there is no good tool
to find out un-intended access to valid area. This kind of problem is really
hard to solve so this tool would be very useful.

This tool doesn't automatically catch a problem. Manual runtime configuration
to specify the target object is required.

Note that there was a similar attempt for the debugging overwritten problem
however it requires manual code modifying and recompile.

http://lkml.kernel.org/r/<20171117223043.7277-1-wen.gang.w...@oracle.com>

To get more information about vchecker, please see a documention at
the last patch.

Patchset can also be available at

https://github.com/JoonsooKim/linux/tree/vchecker-master-v1.0-next-20171122

Enjoy it.

Thanks.

Joonsoo Kim (14):
  mm/kasan: make some kasan functions global
  vchecker: introduce the valid access checker
  vchecker: mark/unmark the shadow of the allocated objects
  vchecker: prepare per object memory for vchecker
  vchecker: store/report callstack of value writer
  lib/stackdepot: extend stackdepot API to support per-user stackdepot
  vchecker: consistently exclude vchecker's stacktrace
  vchecker: fix 'remove' handling on callstack checker
  mm/vchecker: support inline KASAN build
  mm/vchecker: make callstack depth configurable
  mm/vchecker: pass allocation caller address to vchecker hook
  mm/vchecker: support allocation caller filter
  lib/vchecker_test: introduce a sample for vchecker test
  doc: add vchecker document

Namhyung Kim (4):
  lib/stackdepot: Add is_new arg to depot_save_stack
  vchecker: Add 'callstack' checker
  vchecker: Support toggle on/off of callstack check
  vchecker: Use __GFP_ATOMIC to save stacktrace

 Documentation/dev-tools/vchecker.rst |  200 +++
 drivers/gpu/drm/drm_mm.c |4 +-
 include/linux/kasan.h|1 +
 include/linux/slab.h |8 +
 include/linux/slab_def.h |3 +
 include/linux/slub_def.h |3 +
 include/linux/stackdepot.h   |   10 +-
 lib/Kconfig.kasan|   21 +
 lib/Makefile |1 +
 lib/stackdepot.c |  126 ++--
 lib/vchecker_test.c  |  117 
 mm/kasan/Makefile|1 +
 mm/kasan/kasan.c |   14 +-
 mm/kasan/kasan.h |3 +
 mm/kasan/report.c|   12 +-
 mm/kasan/vchecker.c  | 1089 ++
 mm/kasan/vchecker.h  |   43 ++
 mm/page_owner.c  |8 +-
 mm/slab.c|   47 +-
 mm/slab.h|   14 +-
 mm/slab_common.c |   25 +
 mm/slub.c|   49 +-
 22 files changed, 1730 insertions(+), 69 deletions(-)
 create mode 100644 Documentation/dev-tools/vchecker.rst
 create mode 100644 lib/vchecker_test.c
 create mode 100644 mm/kasan/vchecker.c
 create mode 100644 mm/kasan/vchecker.h

-- 
2.7.4