Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-18 Thread Andrea Arcangeli
On Tue, Apr 14, 2009 at 03:09:29PM -0700, Andrew Morton wrote: We need a comment here explaining why we can't use the much preferable lock_page(). Why can't we use the much preferable lock_page()? We might but then it'd risk to waste time waiting. It's not worth waiting, we want kksmd to be

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-16 Thread Andrea Arcangeli
On Wed, Apr 15, 2009 at 05:43:03PM -0700, Jeremy Fitzhardinge wrote: Shouldn't that be kmap_atomic's job anyway? Otherwise it would be hard to No because those are full noops in no-highmem kernels. I commented in other email why I think it's safe thanks to the wrprotect + smp tlb flush of the

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-16 Thread Jeremy Fitzhardinge
Andrea Arcangeli wrote: On Wed, Apr 15, 2009 at 05:43:03PM -0700, Jeremy Fitzhardinge wrote: Shouldn't that be kmap_atomic's job anyway? Otherwise it would be hard to No because those are full noops in no-highmem kernels. I commented in other email why I think it's safe thanks to the

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-15 Thread Izik Eidus
Andrew Morton wrote: On Thu, 9 Apr 2009 06:58:41 +0300 Izik Eidus iei...@redhat.com wrote: Confused. In the covering email you indicated that v2 of the patchset had abandoned ioctls and had moved the interface to sysfs. We have abandoned the ioctls that control the ksm behavior (how

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-15 Thread Andrew Morton
On Thu, 16 Apr 2009 01:37:25 +0300 Izik Eidus iei...@redhat.com wrote: Andrew Morton wrote: On Thu, 9 Apr 2009 06:58:41 +0300 Izik Eidus iei...@redhat.com wrote: Confused. In the covering email you indicated that v2 of the patchset had abandoned ioctls and had moved the

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-15 Thread Andrea Arcangeli
On Wed, Apr 15, 2009 at 03:50:58PM -0700, Andrew Morton wrote: an optional thing and can even be modprobed, that doesn't work. And having a driver in mm/ which can be modprobed is kinda neat. Agreed. I think madvise with all its vma split requirements and ksm-unregistering invoked at vma

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-15 Thread Jeremy Fitzhardinge
Andrew Morton wrote: +static pte_t *get_pte(struct mm_struct *mm, unsigned long addr) +{ + pgd_t *pgd; + pud_t *pud; + pmd_t *pmd; + pte_t *ptep = NULL; + + pgd = pgd_offset(mm, addr); + if (!pgd_present(*pgd)) + goto out; + + pud =

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-15 Thread Izik Eidus
Jeremy Fitzhardinge wrote: Andrew Morton wrote: +static pte_t *get_pte(struct mm_struct *mm, unsigned long addr) +{ +pgd_t *pgd; +pud_t *pud; +pmd_t *pmd; +pte_t *ptep = NULL; + +pgd = pgd_offset(mm, addr); +if (!pgd_present(*pgd)) +goto out; + +pud =

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-14 Thread Andrew Morton
On Thu, 9 Apr 2009 06:58:41 +0300 Izik Eidus iei...@redhat.com wrote: Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to

[PATCH 4/4] add ksm kernel shared memory driver.

2009-04-08 Thread Izik Eidus
Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to change them. Ksm is used for cases where using fork() is not suitable, one of

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-06 Thread Izik Eidus
Andrey Panin wrote: On 094, 04 04, 2009 at 05:35:22PM +0300, Izik Eidus wrote: SNIP +static inline u32 calc_checksum(struct page *page) +{ + u32 checksum; + void *addr = kmap_atomic(page, KM_USER0); + checksum = jhash(addr, PAGE_SIZE, 17); Why jhash2() is not used

[PATCH 4/4] add ksm kernel shared memory driver.

2009-04-04 Thread Izik Eidus
Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to change them. Ksm is used for cases where using fork() is not suitable, one of

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-02 Thread Avi Kivity
Anthony Liguori wrote: I'm often afraid of what sort of bugs we'd uncover in kvm if we passed the fds around via SCM_RIGHTS and started poking around :-/ kvm checks the mm doesn't change underneath. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-02 Thread Andrea Arcangeli
On Wed, Apr 01, 2009 at 09:36:31PM -0500, Anthony Liguori wrote: on this behavior to unregister memory regions, you could potentially have badness happen in the kernel if ksm attempted to access an invalid memory region. How could you possibly come to this conclusion? If badness could ever

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-02 Thread Izik Eidus
Anthony Liguori wrote: Chris Wright wrote: * Anthony Liguori (anth...@codemonkey.ws) wrote: The ioctl() interface is quite bad for what you're doing. You're telling the kernel extra information about a VA range in userspace. That's what madvise is for. You're tweaking simple

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Izik Eidus
KAMEZAWA Hiroyuki wrote: On Tue, 31 Mar 2009 15:21:53 +0300 Izik Eidus iei...@redhat.com wrote: kpage is actually what going to be KsmPage - the shared page... Right now this pages are not swappable..., after ksm will be merged we will make this pages swappable as well...

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Izik Eidus
Anthony Liguori wrote: Andrea Arcangeli wrote: On Tue, Mar 31, 2009 at 10:54:57AM -0500, Anthony Liguori wrote: You can still disable ksm and simply return ENOSYS for the MADV_ flag. You Anthony, the biggest problem about madvice() is that it is a real system call api, i wouldnt

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Anthony Liguori
Izik Eidus wrote: Anthony, the biggest problem about madvice() is that it is a real system call api, i wouldnt want in that stage of ksm commit into api changes of linux... The ioctl itself is restricting, madvice is much more..., Can we draft this issue to after ksm is merged, and after all

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Chris Wright
* Anthony Liguori (anth...@codemonkey.ws) wrote: You can't change ABIs after something is merged or you break userspace. So you need to figure out the right ABI first. Absolutely. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Chris Wright
* Anthony Liguori (anth...@codemonkey.ws) wrote: The ioctl() interface is quite bad for what you're doing. You're telling the kernel extra information about a VA range in userspace. That's what madvise is for. You're tweaking simple read/write values of kernel infrastructure. That's

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-04-01 Thread Anthony Liguori
Chris Wright wrote: * Anthony Liguori (anth...@codemonkey.ws) wrote: The ioctl() interface is quite bad for what you're doing. You're telling the kernel extra information about a VA range in userspace. That's what madvise is for. You're tweaking simple read/write values of kernel

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Izik Eidus
KAMEZAWA Hiroyuki wrote: On Tue, 31 Mar 2009 02:59:20 +0300 Izik Eidus iei...@redhat.com wrote: Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Izik Eidus
Anthony Liguori wrote: Izik Eidus wrote: Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to change them. Ksm is used for cases

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Anthony Liguori
Izik Eidus wrote: I belive using ioctl for registering memory of applications make it easier Yes, I completely agree. Ksm doesnt have any complicated API that would benefit from sysfs (beside adding more complexity) That is, the KSM_START_STOP_KTHREAD part, not necessarily the rest

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
On Tue, Mar 31, 2009 at 08:31:31AM -0500, Anthony Liguori wrote: You could drop KSM_START_STOP_KTHREAD and KSM_GET_INFO_KTHREAD altogether, and introduce a sysfs hierarchy: /sysfs/some/path/ksm/{enable,pages_to_scan,sleep_time} Introducing a sysfs hierarchy sounds a bit of overkill. the

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Anthony Liguori
Andrea Arcangeli wrote: the ability to disable KSM. That seems like a security concern to me since registering a memory region ought to be an unprivileged action whereas enabling/disabling KSM ought to be a privileged action. sysfs files would then only be writeable by admin, so if we

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
On Tue, Mar 31, 2009 at 09:37:17AM -0500, Anthony Liguori wrote: In the very least, if you insist on not using sysfs, you should have a separate character device that's used for control (like /dev/ksmctl). I'm fine to use sysfs that's not the point, if you've to add a ksmctl device, then sysfs

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Anthony Liguori
Andrea Arcangeli wrote: On Tue, Mar 31, 2009 at 09:37:17AM -0500, Anthony Liguori wrote: In the very least, if you insist on not using sysfs, you should have a separate character device that's used for control (like /dev/ksmctl). I'm fine to use sysfs that's not the point, if you've

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
On Tue, Mar 31, 2009 at 10:09:24AM -0500, Anthony Liguori wrote: I don't think the registering of ram should be done via sysfs. That would be a pretty bad interface IMHO. But I do think the functionality that ksmctl provides along with the security issues I mentioned earlier really

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Anthony Liguori
Andrea Arcangeli wrote: On Tue, Mar 31, 2009 at 10:09:24AM -0500, Anthony Liguori wrote: I don't think the registering of ram should be done via sysfs. That would be a pretty bad interface IMHO. But I do think the functionality that ksmctl provides along with the security issues I

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
On Tue, Mar 31, 2009 at 10:54:57AM -0500, Anthony Liguori wrote: You can still disable ksm and simply return ENOSYS for the MADV_ flag. You -EINVAL if something, -ENOSYS would tell userland that it shall stop trying to use madvise, including the other MADV_ too. could even keep it as a

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Anthony Liguori
Andrea Arcangeli wrote: On Tue, Mar 31, 2009 at 10:54:57AM -0500, Anthony Liguori wrote: You can still disable ksm and simply return ENOSYS for the MADV_ flag. You -EINVAL if something, -ENOSYS would tell userland that it shall stop trying to use madvise, including the other MADV_

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
On Tue, Mar 31, 2009 at 11:51:14AM -0500, Anthony Liguori wrote: You have two things here. CONFIG_MEM_SHARABLE and CONFIG_KSM. CONFIG_MEM_SHARABLE cannot be a module. If it's set to =n, then madvise(MADV_SHARABLE) == -ENOSYS. Where the part that -ENOSYS tell userland madvise syscall table

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread Andrea Arcangeli
Hello, I attach below some benchmark of the new ksm tree algorithm, showing ksm performance in best and worst case scenarios. --- Here a program ksmpages.c that tries to create the worst case scenario for the ksm tree algorithm.

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-31 Thread KAMEZAWA Hiroyuki
On Tue, 31 Mar 2009 15:21:53 +0300 Izik Eidus iei...@redhat.com wrote: kpage is actually what going to be KsmPage - the shared page... Right now this pages are not swappable..., after ksm will be merged we will make this pages swappable as well... sure. If so, please - show the

[PATCH 4/4] add ksm kernel shared memory driver.

2009-03-30 Thread Izik Eidus
Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to change them. Ksm is used for cases where using fork() is not suitable, one of

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-30 Thread Anthony Liguori
Izik Eidus wrote: Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to change them. Ksm is used for cases where using fork() is

Re: [PATCH 4/4] add ksm kernel shared memory driver.

2009-03-30 Thread KAMEZAWA Hiroyuki
On Tue, 31 Mar 2009 02:59:20 +0300 Izik Eidus iei...@redhat.com wrote: Ksm is driver that allow merging identical pages between one or more applications in way unvisible to the application that use it. Pages that are merged are marked as readonly and are COWed when any application try to