Re: [KVM-AUTOTEST PATCH] KVM test: Add hugepage variant

[Lukáš Doktor]

Hello Ryan,

see below...

Dne 29.7.2009 16:41, Ryan Harper napsal(a):

* Lucas Meneghel Rodriguesl...@redhat.com  [2009-07-28 22:40]:

This patch adds a small setup script to set up huge memory
pages during the kvm tests execution. Also, added hugepage setup to the
fc8_quick sample.

Signed-off-by: Luká?? Doktorldok...@redhat.com
Signed-off-by: Lucas Meneghel Rodriguesl...@redhat.com


Looks good.  one nit below.

Signed-off-by: Ryan Harperry...@us.ibm.com


---
+
+def get_target_hugepages(self):
+
+Calculate the target number of hugepages for testing purposes.
+
+if self.vms  self.max_vms:
+self.vms = self.max_vms
+vmsm = (self.vms * self.mem) + (self.vms * 64)


Nit: Maybe a comment about the fudge factor being added in?


It's qemu-kvm overhead. Should I change the patch or is this explanation 
sufficient?


Thanks for the feedback,
bye, Lukáš
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [KVM-AUTOTEST PATCH] KVM test: Add hugepage variant

[Lukáš Doktor]

Dne 29.7.2009 05:40, Lucas Meneghel Rodrigues napsal(a):

This patch adds a small setup script to set up huge memory
pages during the kvm tests execution. Also, added hugepage setup to the
fc8_quick sample.

Signed-off-by: Lukáš Doktorldok...@redhat.com
Signed-off-by: Lucas Meneghel Rodriguesl...@redhat.com
---
  client/tests/kvm/kvm_tests.cfg.sample |8 +++
  client/tests/kvm/kvm_vm.py|   11 +++
  client/tests/kvm/scripts/hugepage.py  |  109 +
  3 files changed, 128 insertions(+), 0 deletions(-)
  create mode 100644 client/tests/kvm/scripts/hugepage.py

diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
b/client/tests/kvm/kvm_tests.cfg.sample
index 2d75a66..7cd12cb 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -587,6 +587,13 @@ variants:


  variants:
+- @kvm_smallpages:
+- kvm_hugepages:
+pre_command = /usr/bin/python scripts/hugepage.py /mnt/kvm_hugepage
+extra_params +=  -mem-path /mnt/kvm_hugepage
+
+
+variants:
  - @basic:
  only Fedora Windows
  - @full:
@@ -598,6 +605,7 @@ variants:
  only Fedora.8.32
  only install setup boot shutdown
  only rtl8139
+only kvm_hugepages
  - @sample1:
  only qcow2
  only ide
diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index d96b359..eba9b84 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -397,6 +397,17 @@ class VM:
  self.destroy()
  return False

+# Get the output so far, to see if we have any problems with
+# hugepage setup.
+output = self.process.get_output()
+
+if alloc_mem_area in output:
+logging.error(Could not allocate hugepage memory; 
+  qemu command:\n%s % qemu_command)
+logging.error(Output: + kvm_utils.format_str_for_message(
+  self.process.get_output()))
+return False
+
  logging.debug(VM appears to be alive with PID %d,
self.process.get_pid())
  return True
diff --git a/client/tests/kvm/scripts/hugepage.py 
b/client/tests/kvm/scripts/hugepage.py
new file mode 100644
index 000..dc36da4
--- /dev/null
+++ b/client/tests/kvm/scripts/hugepage.py
@@ -0,0 +1,109 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+import os, sys, time
+
+
+Simple script to allocate enough hugepages for KVM testing purposes.
+
+
+class HugePageError(Exception):
+
+Simple wrapper for the builtin Exception class.
+
+pass
+
+
+class HugePage:
+def __init__(self, hugepage_path=None):
+
+Gets environment variable values and calculates the target number
+of huge memory pages.
+
+@param hugepage_path: Path where to mount hugetlbfs path, if not
+yet configured.
+
+self.vms = len(os.environ['KVM_TEST_vms'].split())
+self.mem = int(os.environ['KVM_TEST_mem'])
+try:
+self.max_vms = int(os.environ['KVM_TEST_max_vms'])
+except KeyError:
+self.max_vms = 0
+
+if hugepage_path:
+self.hugepage_path = hugepage_path
+else:
+self.hugepage_path = '/mnt/kvm_hugepage'
+
+self.hugepage_size = self.get_hugepage_size()
+self.target_hugepages = self.get_target_hugepages()
+
+
+def get_hugepage_size(self):
+
+Get the current system setting for huge memory page size.
+
+meminfo = open('/proc/meminfo', 'r').readlines()
+huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
+try:
+return int(huge_line_list[0].split()[1])
+except ValueError, e:
+raise HugePageError(Could not get huge page size setting from 
+/proc/meminfo: %s % e)
+
+
+def get_target_hugepages(self):
+
+Calculate the target number of hugepages for testing purposes.
+
+if self.vms  self.max_vms:
+self.vms = self.max_vms
+vmsm = (self.vms * self.mem) + (self.vms * 64)
+return int(vmsm * 1024 / self.hugepage_size)
+
+
+def set_hugepages(self):
+
+Sets the hugepage limit to the target hugepage value calculated.
+
+hugepage_cfg = open(/proc/sys/vm/nr_hugepages, r+)
+hp = hugepage_cfg.readline()
+while int(hp)  self.target_hugepages:
+loop_hp = hp
+hugepage_cfg.write(str(self.target_hugepages))
+hugepage_cfg.flush()
+hugepage_cfg.seek(0)
+hp = int(hugepage_cfg.readline())
+if loop_hp == hp:
+raise HugePageError(Cannot set the kernel hugepage setting 
+to the target value of %d hugepages. %
+

Re: [Autotest] [KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Lucas Meneghel Rodrigues]

On Mon, Jul 27, 2009 at 6:10 PM, Lucas Meneghel Rodriguesl...@redhat.com 
wrote:
 This patch adds a small setup script to set up huge memory
 pages during the kvm tests execution. Also, added hugepage setup to the
 fc8_quick sample.

Applied.

 Signed-off-by: Lukáš Doktor ldok...@redhat.com
 Signed-off-by: Lucas Meneghel Rodrigues l...@redhat.com

 ---
  client/tests/kvm/kvm_tests.cfg.sample |    6 ++
  client/tests/kvm/kvm_vm.py            |   11 +++
  client/tests/kvm/scripts/hugepage.py  |  110 
 +
  3 files changed, 127 insertions(+), 0 deletions(-)
  create mode 100644 client/tests/kvm/scripts/hugepage.py

 diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
 b/client/tests/kvm/kvm_tests.cfg.sample
 index 2d75a66..4a6a174 100644
 --- a/client/tests/kvm/kvm_tests.cfg.sample
 +++ b/client/tests/kvm/kvm_tests.cfg.sample
 @@ -585,6 +585,11 @@ variants:
         only default
         image_format = raw

 +variants:
 +    - @kvm_smallpages:
 +    - kvm_hugepages:
 +        pre_command = /usr/bin/python scripts/hugepage.py
 +

  variants:
     - @basic:
 @@ -598,6 +603,7 @@ variants:
         only Fedora.8.32
         only install setup boot shutdown
         only rtl8139
 +        only kvm_hugepages
     - @sample1:
         only qcow2
         only ide
 diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
 index d96b359..eba9b84 100644
 --- a/client/tests/kvm/kvm_vm.py
 +++ b/client/tests/kvm/kvm_vm.py
 @@ -397,6 +397,17 @@ class VM:
                 self.destroy()
                 return False

 +            # Get the output so far, to see if we have any problems with
 +            # hugepage setup.
 +            output = self.process.get_output()
 +
 +            if alloc_mem_area in output:
 +                logging.error(Could not allocate hugepage memory; 
 +                              qemu command:\n%s % qemu_command)
 +                logging.error(Output: + kvm_utils.format_str_for_message(
 +                              self.process.get_output()))
 +                return False
 +
             logging.debug(VM appears to be alive with PID %d,
                           self.process.get_pid())
             return True
 diff --git a/client/tests/kvm/scripts/hugepage.py 
 b/client/tests/kvm/scripts/hugepage.py
 new file mode 100644
 index 000..9bc4194
 --- /dev/null
 +++ b/client/tests/kvm/scripts/hugepage.py
 @@ -0,0 +1,110 @@
 +#!/usr/bin/python
 +# -*- coding: utf-8 -*-
 +import os, sys, time
 +
 +
 +Simple script to allocate enough hugepages for KVM testing purposes.
 +
 +
 +class HugePageError(Exception):
 +    
 +    Simple wrapper for the builtin Exception class.
 +    
 +    pass
 +
 +
 +class HugePage:
 +    def __init__(self, hugepage_path=None):
 +        
 +        Gets environment variable values and calculates the target number
 +        of huge memory pages.
 +
 +       �...@param hugepage_path: Path where to mount hugetlbfs path, if not
 +                yet configured.
 +        
 +        self.vms = len(os.environ['KVM_TEST_vms'].split())
 +        self.mem = int(os.environ['KVM_TEST_mem'])
 +        try:
 +            self.max_vms = int(os.environ['KVM_TEST_max_vms'])
 +        except KeyError:
 +            self.max_vms = 0
 +        if hugepage_path:
 +            self.hugepage_path = hugepage_path
 +        else:
 +            self.hugepage_path = '/mnt/kvm_hugepage'
 +        self.hugepage_size = self.get_hugepage_size()
 +        self.target_hugepages = self.get_target_hugepages()
 +
 +
 +    def get_hugepage_size(self):
 +        
 +        Get the current system setting for huge memory page size.
 +        
 +        meminfo = open('/proc/meminfo', 'r').readlines()
 +        huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
 +        try:
 +            return int(huge_line_list[0].split()[1])
 +        except ValueError, e:
 +            raise HugePageError(Could not get huge page size setting from 
 +                                /proc/meminfo: %s % e)
 +
 +
 +    def get_target_hugepages(self):
 +        
 +        Calculate the target number of hugepages for testing purposes.
 +        
 +        if self.vms  self.max_vms:
 +            self.vms = self.max_vms
 +        vmsm = (self.vms * self.mem) + (self.vms * 64)
 +        return int(vmsm * 1024 / self.hugepage_size)
 +
 +
 +    def set_hugepages(self):
 +        
 +        Sets the hugepage limit to the target hugepage value calculated.
 +        
 +        hugepage_cfg = open(/proc/sys/vm/nr_hugepages, r+)
 +        hp = hugepage_cfg.readline()
 +        while int(hp)  self.target_hugepages:
 +            loop_hp = hp
 +            hugepage_cfg.write(str(self.target_hugepages))
 +            hugepage_cfg.flush()
 +            time.sleep(5)
 +            hugepage_cfg.seek(0)
 +            hp = int(hugepage_cfg.readline())
 +            if loop_hp == hp:
 +                raise HugePageError(Cannot set the kernel hugepage setting 
 +            

Re: [KVM-AUTOTEST PATCH] KVM test: Add hugepage variant

[Ryan Harper]

* Lucas Meneghel Rodrigues l...@redhat.com [2009-07-28 22:40]:
 This patch adds a small setup script to set up huge memory
 pages during the kvm tests execution. Also, added hugepage setup to the
 fc8_quick sample.
 
 Signed-off-by: Luká?? Doktor ldok...@redhat.com
 Signed-off-by: Lucas Meneghel Rodrigues l...@redhat.com

Looks good.  one nit below.

Signed-off-by: Ryan Harper ry...@us.ibm.com

 ---
  client/tests/kvm/kvm_tests.cfg.sample |8 +++
  client/tests/kvm/kvm_vm.py|   11 +++
  client/tests/kvm/scripts/hugepage.py  |  109 
 +
  3 files changed, 128 insertions(+), 0 deletions(-)
  create mode 100644 client/tests/kvm/scripts/hugepage.py
 
 diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
 b/client/tests/kvm/kvm_tests.cfg.sample
 index 2d75a66..7cd12cb 100644
 --- a/client/tests/kvm/kvm_tests.cfg.sample
 +++ b/client/tests/kvm/kvm_tests.cfg.sample
 @@ -587,6 +587,13 @@ variants:
 
 
  variants:
 +- @kvm_smallpages:
 +- kvm_hugepages:
 +pre_command = /usr/bin/python scripts/hugepage.py /mnt/kvm_hugepage
 +extra_params +=  -mem-path /mnt/kvm_hugepage
 +
 +
 +variants:
  - @basic:
  only Fedora Windows
  - @full:
 @@ -598,6 +605,7 @@ variants:
  only Fedora.8.32
  only install setup boot shutdown
  only rtl8139
 +only kvm_hugepages
  - @sample1:
  only qcow2
  only ide
 diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
 index d96b359..eba9b84 100644
 --- a/client/tests/kvm/kvm_vm.py
 +++ b/client/tests/kvm/kvm_vm.py
 @@ -397,6 +397,17 @@ class VM:
  self.destroy()
  return False
 
 +# Get the output so far, to see if we have any problems with
 +# hugepage setup.
 +output = self.process.get_output()
 +
 +if alloc_mem_area in output:
 +logging.error(Could not allocate hugepage memory; 
 +  qemu command:\n%s % qemu_command)
 +logging.error(Output: + kvm_utils.format_str_for_message(
 +  self.process.get_output()))
 +return False
 +
  logging.debug(VM appears to be alive with PID %d,
self.process.get_pid())
  return True
 diff --git a/client/tests/kvm/scripts/hugepage.py 
 b/client/tests/kvm/scripts/hugepage.py
 new file mode 100644
 index 000..dc36da4
 --- /dev/null
 +++ b/client/tests/kvm/scripts/hugepage.py
 @@ -0,0 +1,109 @@
 +#!/usr/bin/python
 +# -*- coding: utf-8 -*-
 +import os, sys, time
 +
 +
 +Simple script to allocate enough hugepages for KVM testing purposes.
 +
 +
 +class HugePageError(Exception):
 +
 +Simple wrapper for the builtin Exception class.
 +
 +pass
 +
 +
 +class HugePage:
 +def __init__(self, hugepage_path=None):
 +
 +Gets environment variable values and calculates the target number
 +of huge memory pages.
 +
 +@param hugepage_path: Path where to mount hugetlbfs path, if not
 +yet configured.
 +
 +self.vms = len(os.environ['KVM_TEST_vms'].split())
 +self.mem = int(os.environ['KVM_TEST_mem'])
 +try:
 +self.max_vms = int(os.environ['KVM_TEST_max_vms'])
 +except KeyError:
 +self.max_vms = 0
 +
 +if hugepage_path:
 +self.hugepage_path = hugepage_path
 +else:
 +self.hugepage_path = '/mnt/kvm_hugepage'
 +
 +self.hugepage_size = self.get_hugepage_size()
 +self.target_hugepages = self.get_target_hugepages()
 +
 +
 +def get_hugepage_size(self):
 +
 +Get the current system setting for huge memory page size.
 +
 +meminfo = open('/proc/meminfo', 'r').readlines()
 +huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
 +try:
 +return int(huge_line_list[0].split()[1])
 +except ValueError, e:
 +raise HugePageError(Could not get huge page size setting from 
 +/proc/meminfo: %s % e)
 +
 +
 +def get_target_hugepages(self):
 +
 +Calculate the target number of hugepages for testing purposes.
 +
 +if self.vms  self.max_vms:
 +self.vms = self.max_vms
 +vmsm = (self.vms * self.mem) + (self.vms * 64)

Nit: Maybe a comment about the fudge factor being added in?

 +return int(vmsm * 1024 / self.hugepage_size)
 +
 +
 +def set_hugepages(self):
 +
 +Sets the hugepage limit to the target hugepage value calculated.
 +
 +hugepage_cfg = open(/proc/sys/vm/nr_hugepages, r+)
 +hp = hugepage_cfg.readline()
 +while int(hp)  self.target_hugepages:
 +loop_hp = hp
 +hugepage_cfg.write(str(self.target_hugepages))
 +hugepage_cfg.flush()
 +

Re: [KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Lukáš Doktor]

Yes, this looks more pythonish and actually better than my version. I'm 
missing only one thing, extra_params +=  -mem-path /mnt/hugepage down 
in configuration (see below).


This cause problem with predefined mount point, because it needs to be 
the same in extra_params and python script.


Dne 27.7.2009 23:10, Lucas Meneghel Rodrigues napsal(a):

This patch adds a small setup script to set up huge memory
pages during the kvm tests execution. Also, added hugepage setup to the
fc8_quick sample.

Signed-off-by: LukĂĄĹĄ Doktorldok...@redhat.com
Signed-off-by: Lucas Meneghel Rodriguesl...@redhat.com

---
  client/tests/kvm/kvm_tests.cfg.sample |6 ++
  client/tests/kvm/kvm_vm.py|   11 +++
  client/tests/kvm/scripts/hugepage.py  |  110 +
  3 files changed, 127 insertions(+), 0 deletions(-)
  create mode 100644 client/tests/kvm/scripts/hugepage.py

diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
b/client/tests/kvm/kvm_tests.cfg.sample
index 2d75a66..4a6a174 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -585,6 +585,11 @@ variants:
  only default
  image_format = raw

+variants:
+- @kvm_smallpages:
+- kvm_hugepages:
+pre_command = /usr/bin/python scripts/hugepage.py


+extra_params +=  -mem-path /mnt/hugepage

# ^^Tells qemu to allocate guest memory as hugepage

I'd rather have this part of cfg look like this:
variants:
- @kvm_smallpages:
- kvm_hugepages:
pre_command = /usr/bin/python scripts/hugepage.py 
/mnt/hugepage
extra_params +=  -mem-path /mnt/hugepage

because this way it's more clear the relation between the constants. (it 
doesn't changes the script itself)



+

  variants:
  - @basic:
@@ -598,6 +603,7 @@ variants:
  only Fedora.8.32
  only install setup boot shutdown
  only rtl8139
+only kvm_hugepages
  - @sample1:
  only qcow2
  only ide
diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index d96b359..eba9b84 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -397,6 +397,17 @@ class VM:
  self.destroy()
  return False

+# Get the output so far, to see if we have any problems with
+# hugepage setup.
+output = self.process.get_output()
+
+if alloc_mem_area in output:
+logging.error(Could not allocate hugepage memory; 
+  qemu command:\n%s % qemu_command)
+logging.error(Output: + kvm_utils.format_str_for_message(
+  self.process.get_output()))
+return False
+
  logging.debug(VM appears to be alive with PID %d,
self.process.get_pid())
  return True
diff --git a/client/tests/kvm/scripts/hugepage.py 
b/client/tests/kvm/scripts/hugepage.py
new file mode 100644
index 000..9bc4194
--- /dev/null
+++ b/client/tests/kvm/scripts/hugepage.py
@@ -0,0 +1,110 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+import os, sys, time
+
+
+Simple script to allocate enough hugepages for KVM testing purposes.
+
+
+class HugePageError(Exception):
+
+Simple wrapper for the builtin Exception class.
+
+pass
+
+
+class HugePage:
+def __init__(self, hugepage_path=None):
+
+Gets environment variable values and calculates the target number
+of huge memory pages.
+
+@param hugepage_path: Path where to mount hugetlbfs path, if not
+yet configured.
+
+self.vms = len(os.environ['KVM_TEST_vms'].split())
+self.mem = int(os.environ['KVM_TEST_mem'])
+try:
+self.max_vms = int(os.environ['KVM_TEST_max_vms'])
+except KeyError:
+self.max_vms = 0
+if hugepage_path:
+self.hugepage_path = hugepage_path
+else:
+self.hugepage_path = '/mnt/kvm_hugepage'
+self.hugepage_size = self.get_hugepage_size()
+self.target_hugepages = self.get_target_hugepages()
+
+
+def get_hugepage_size(self):
+
+Get the current system setting for huge memory page size.
+
+meminfo = open('/proc/meminfo', 'r').readlines()
+huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
+try:
+return int(huge_line_list[0].split()[1])
+except ValueError, e:
+raise HugePageError(Could not get huge page size setting from 
+/proc/meminfo: %s % e)
+
+
+def get_target_hugepages(self):
+
+Calculate the target number of hugepages for testing purposes.
+
+if self.vms  self.max_vms:
+self.vms = self.max_vms
+vmsm = (self.vms * self.mem) + (self.vms * 64)
+return int(vmsm * 1024 / self.hugepage_size)
+

Re: [KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Ryan Harper]

* Luk?? Doktor ldok...@redhat.com [2009-07-28 08:22]:
 Yes, this looks more pythonish and actually better than my version. I'm 
 missing only one thing, extra_params +=  -mem-path /mnt/hugepage down 
 in configuration (see below).

Don't we also need to inspect the qemu binary to determine if it's one
of the few releases that used -mempath instead of -mem-path ?  Or are we
ignoring those?


 
 This cause problem with predefined mount point, because it needs to be 
 the same in extra_params and python script.
 
 Dne 27.7.2009 23:10, Lucas Meneghel Rodrigues napsal(a):
 This patch adds a small setup script to set up huge memory
 pages during the kvm tests execution. Also, added hugepage setup to the
 fc8_quick sample.
 
 Signed-off-by: Luk Doktorldok...@redhat.com
 Signed-off-by: Lucas Meneghel Rodriguesl...@redhat.com
 
 ---
   client/tests/kvm/kvm_tests.cfg.sample |6 ++
   client/tests/kvm/kvm_vm.py|   11 +++
   client/tests/kvm/scripts/hugepage.py  |  110 
   +
   3 files changed, 127 insertions(+), 0 deletions(-)
   create mode 100644 client/tests/kvm/scripts/hugepage.py
 
 diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
 b/client/tests/kvm/kvm_tests.cfg.sample
 index 2d75a66..4a6a174 100644
 --- a/client/tests/kvm/kvm_tests.cfg.sample
 +++ b/client/tests/kvm/kvm_tests.cfg.sample
 @@ -585,6 +585,11 @@ variants:
   only default
   image_format = raw
 
 +variants:
 +- @kvm_smallpages:
 +- kvm_hugepages:
 +pre_command = /usr/bin/python scripts/hugepage.py
 
 +extra_params +=  -mem-path /mnt/hugepage
 
 # ^^Tells qemu to allocate guest memory as hugepage
 
 I'd rather have this part of cfg look like this:
 variants:
   - @kvm_smallpages:
   - kvm_hugepages:
   pre_command = /usr/bin/python scripts/hugepage.py 
   /mnt/hugepage
   extra_params +=  -mem-path /mnt/hugepage
 
 because this way it's more clear the relation between the constants. (it 
 doesn't changes the script itself)
 
 +
 
   variants:
   - @basic:
 @@ -598,6 +603,7 @@ variants:
   only Fedora.8.32
   only install setup boot shutdown
   only rtl8139
 +only kvm_hugepages
   - @sample1:
   only qcow2
   only ide
 diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
 index d96b359..eba9b84 100644
 --- a/client/tests/kvm/kvm_vm.py
 +++ b/client/tests/kvm/kvm_vm.py
 @@ -397,6 +397,17 @@ class VM:
   self.destroy()
   return False
 
 +# Get the output so far, to see if we have any problems with
 +# hugepage setup.
 +output = self.process.get_output()
 +
 +if alloc_mem_area in output:
 +logging.error(Could not allocate hugepage memory; 
 +  qemu command:\n%s % qemu_command)
 +logging.error(Output: + 
 kvm_utils.format_str_for_message(
 +  self.process.get_output()))
 +return False
 +
   logging.debug(VM appears to be alive with PID %d,
 self.process.get_pid())
   return True
 diff --git a/client/tests/kvm/scripts/hugepage.py 
 b/client/tests/kvm/scripts/hugepage.py
 new file mode 100644
 index 000..9bc4194
 --- /dev/null
 +++ b/client/tests/kvm/scripts/hugepage.py
 @@ -0,0 +1,110 @@
 +#!/usr/bin/python
 +# -*- coding: utf-8 -*-
 +import os, sys, time
 +
 +
 +Simple script to allocate enough hugepages for KVM testing purposes.
 +
 +
 +class HugePageError(Exception):
 +
 +Simple wrapper for the builtin Exception class.
 +
 +pass
 +
 +
 +class HugePage:
 +def __init__(self, hugepage_path=None):
 +
 +Gets environment variable values and calculates the target number
 +of huge memory pages.
 +
 +@param hugepage_path: Path where to mount hugetlbfs path, if not
 +yet configured.
 +
 +self.vms = len(os.environ['KVM_TEST_vms'].split())
 +self.mem = int(os.environ['KVM_TEST_mem'])
 +try:
 +self.max_vms = int(os.environ['KVM_TEST_max_vms'])
 +except KeyError:
 +self.max_vms = 0
 +if hugepage_path:
 +self.hugepage_path = hugepage_path
 +else:
 +self.hugepage_path = '/mnt/kvm_hugepage'
 +self.hugepage_size = self.get_hugepage_size()
 +self.target_hugepages = self.get_target_hugepages()
 +
 +
 +def get_hugepage_size(self):
 +
 +Get the current system setting for huge memory page size.
 +
 +meminfo = open('/proc/meminfo', 'r').readlines()
 +huge_line_list = [h for h in meminfo if 
 h.startswith(Hugepagesize)]
 +try:
 +return int(huge_line_list[0].split()[1])
 +except ValueError, e:
 +raise HugePageError(Could not get huge page size setting 
 from 
 +  

Re: [Autotest] [KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Lucas Meneghel Rodrigues]

On Tue, Jul 28, 2009 at 10:30 AM, Ryan Harperry...@us.ibm.com wrote:
 * Luk?? Doktor ldok...@redhat.com [2009-07-28 08:22]:
 Yes, this looks more pythonish and actually better than my version. I'm
 missing only one thing, extra_params +=  -mem-path /mnt/hugepage down
 in configuration (see below).

 Don't we also need to inspect the qemu binary to determine if it's one
 of the few releases that used -mempath instead of -mem-path ?  Or are we
 ignoring those?

I am thinking here whether it's worth to add custom logic to kvm_vm
code to deal with this 'corner case'...
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[KVM-AUTOTEST PATCH] KVM test: Add hugepage variant

[Lucas Meneghel Rodrigues]

This patch adds a small setup script to set up huge memory
pages during the kvm tests execution. Also, added hugepage setup to the
fc8_quick sample.

Signed-off-by: Lukáš Doktor ldok...@redhat.com
Signed-off-by: Lucas Meneghel Rodrigues l...@redhat.com
---
 client/tests/kvm/kvm_tests.cfg.sample |8 +++
 client/tests/kvm/kvm_vm.py|   11 +++
 client/tests/kvm/scripts/hugepage.py  |  109 +
 3 files changed, 128 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/scripts/hugepage.py

diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
b/client/tests/kvm/kvm_tests.cfg.sample
index 2d75a66..7cd12cb 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -587,6 +587,13 @@ variants:
 
 
 variants:
+- @kvm_smallpages:
+- kvm_hugepages:
+pre_command = /usr/bin/python scripts/hugepage.py /mnt/kvm_hugepage
+extra_params +=  -mem-path /mnt/kvm_hugepage
+
+
+variants:
 - @basic:
 only Fedora Windows
 - @full:
@@ -598,6 +605,7 @@ variants:
 only Fedora.8.32
 only install setup boot shutdown
 only rtl8139
+only kvm_hugepages
 - @sample1:
 only qcow2
 only ide
diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index d96b359..eba9b84 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -397,6 +397,17 @@ class VM:
 self.destroy()
 return False
 
+# Get the output so far, to see if we have any problems with
+# hugepage setup.
+output = self.process.get_output()
+
+if alloc_mem_area in output:
+logging.error(Could not allocate hugepage memory; 
+  qemu command:\n%s % qemu_command)
+logging.error(Output: + kvm_utils.format_str_for_message(
+  self.process.get_output()))
+return False
+
 logging.debug(VM appears to be alive with PID %d,
   self.process.get_pid())
 return True
diff --git a/client/tests/kvm/scripts/hugepage.py 
b/client/tests/kvm/scripts/hugepage.py
new file mode 100644
index 000..dc36da4
--- /dev/null
+++ b/client/tests/kvm/scripts/hugepage.py
@@ -0,0 +1,109 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+import os, sys, time
+
+
+Simple script to allocate enough hugepages for KVM testing purposes.
+
+
+class HugePageError(Exception):
+
+Simple wrapper for the builtin Exception class.
+
+pass
+
+
+class HugePage:
+def __init__(self, hugepage_path=None):
+
+Gets environment variable values and calculates the target number
+of huge memory pages.
+
+@param hugepage_path: Path where to mount hugetlbfs path, if not
+yet configured.
+
+self.vms = len(os.environ['KVM_TEST_vms'].split())
+self.mem = int(os.environ['KVM_TEST_mem'])
+try:
+self.max_vms = int(os.environ['KVM_TEST_max_vms'])
+except KeyError:
+self.max_vms = 0
+
+if hugepage_path:
+self.hugepage_path = hugepage_path
+else:
+self.hugepage_path = '/mnt/kvm_hugepage'
+
+self.hugepage_size = self.get_hugepage_size()
+self.target_hugepages = self.get_target_hugepages()
+
+
+def get_hugepage_size(self):
+
+Get the current system setting for huge memory page size.
+
+meminfo = open('/proc/meminfo', 'r').readlines()
+huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
+try:
+return int(huge_line_list[0].split()[1])
+except ValueError, e:
+raise HugePageError(Could not get huge page size setting from 
+/proc/meminfo: %s % e)
+
+
+def get_target_hugepages(self):
+
+Calculate the target number of hugepages for testing purposes.
+
+if self.vms  self.max_vms:
+self.vms = self.max_vms
+vmsm = (self.vms * self.mem) + (self.vms * 64)
+return int(vmsm * 1024 / self.hugepage_size)
+
+
+def set_hugepages(self):
+
+Sets the hugepage limit to the target hugepage value calculated.
+
+hugepage_cfg = open(/proc/sys/vm/nr_hugepages, r+)
+hp = hugepage_cfg.readline()
+while int(hp)  self.target_hugepages:
+loop_hp = hp
+hugepage_cfg.write(str(self.target_hugepages))
+hugepage_cfg.flush()
+hugepage_cfg.seek(0)
+hp = int(hugepage_cfg.readline())
+if loop_hp == hp:
+raise HugePageError(Cannot set the kernel hugepage setting 
+to the target value of %d hugepages. %
+self.target_hugepages)
+hugepage_cfg.close()
+
+
+def 

Re: [Autotest] [KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Lucas Meneghel Rodrigues]

On Tue, Jul 28, 2009 at 10:30 AM, Ryan Harperry...@us.ibm.com wrote:
 * Luk?? Doktor ldok...@redhat.com [2009-07-28 08:22]:
 Yes, this looks more pythonish and actually better than my version. I'm
 missing only one thing, extra_params +=  -mem-path /mnt/hugepage down
 in configuration (see below).

 Don't we also need to inspect the qemu binary to determine if it's one
 of the few releases that used -mempath instead of -mem-path ?  Or are we
 ignoring those?

Ok, I amended the original patch with some changes, fixing the
parameter passing, and created another one, that basically keeps
record of the supported qemu options and can replace known misspelling
issues. IMO takes care of the problem, waiting on comments about the
approach!

Thanks!
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[KVM AUTOTEST PATCH] KVM test: Add hugepage variant

[Lucas Meneghel Rodrigues]

This patch adds a small setup script to set up huge memory
pages during the kvm tests execution. Also, added hugepage setup to the
fc8_quick sample.

Signed-off-by: Lukáš Doktor ldok...@redhat.com
Signed-off-by: Lucas Meneghel Rodrigues l...@redhat.com

---
 client/tests/kvm/kvm_tests.cfg.sample |6 ++
 client/tests/kvm/kvm_vm.py|   11 +++
 client/tests/kvm/scripts/hugepage.py  |  110 +
 3 files changed, 127 insertions(+), 0 deletions(-)
 create mode 100644 client/tests/kvm/scripts/hugepage.py

diff --git a/client/tests/kvm/kvm_tests.cfg.sample 
b/client/tests/kvm/kvm_tests.cfg.sample
index 2d75a66..4a6a174 100644
--- a/client/tests/kvm/kvm_tests.cfg.sample
+++ b/client/tests/kvm/kvm_tests.cfg.sample
@@ -585,6 +585,11 @@ variants:
 only default
 image_format = raw
 
+variants:
+- @kvm_smallpages:
+- kvm_hugepages:
+pre_command = /usr/bin/python scripts/hugepage.py
+
 
 variants:
 - @basic:
@@ -598,6 +603,7 @@ variants:
 only Fedora.8.32
 only install setup boot shutdown
 only rtl8139
+only kvm_hugepages
 - @sample1:
 only qcow2
 only ide
diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index d96b359..eba9b84 100644
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
@@ -397,6 +397,17 @@ class VM:
 self.destroy()
 return False
 
+# Get the output so far, to see if we have any problems with
+# hugepage setup.
+output = self.process.get_output()
+
+if alloc_mem_area in output:
+logging.error(Could not allocate hugepage memory; 
+  qemu command:\n%s % qemu_command)
+logging.error(Output: + kvm_utils.format_str_for_message(
+  self.process.get_output()))
+return False
+
 logging.debug(VM appears to be alive with PID %d,
   self.process.get_pid())
 return True
diff --git a/client/tests/kvm/scripts/hugepage.py 
b/client/tests/kvm/scripts/hugepage.py
new file mode 100644
index 000..9bc4194
--- /dev/null
+++ b/client/tests/kvm/scripts/hugepage.py
@@ -0,0 +1,110 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+import os, sys, time
+
+
+Simple script to allocate enough hugepages for KVM testing purposes.
+
+
+class HugePageError(Exception):
+
+Simple wrapper for the builtin Exception class.
+
+pass
+
+
+class HugePage:
+def __init__(self, hugepage_path=None):
+
+Gets environment variable values and calculates the target number
+of huge memory pages.
+
+@param hugepage_path: Path where to mount hugetlbfs path, if not
+yet configured.
+
+self.vms = len(os.environ['KVM_TEST_vms'].split())
+self.mem = int(os.environ['KVM_TEST_mem'])
+try:
+self.max_vms = int(os.environ['KVM_TEST_max_vms'])
+except KeyError:
+self.max_vms = 0
+if hugepage_path:
+self.hugepage_path = hugepage_path
+else:
+self.hugepage_path = '/mnt/kvm_hugepage'
+self.hugepage_size = self.get_hugepage_size()
+self.target_hugepages = self.get_target_hugepages()
+
+
+def get_hugepage_size(self):
+
+Get the current system setting for huge memory page size.
+
+meminfo = open('/proc/meminfo', 'r').readlines()
+huge_line_list = [h for h in meminfo if h.startswith(Hugepagesize)]
+try:
+return int(huge_line_list[0].split()[1])
+except ValueError, e:
+raise HugePageError(Could not get huge page size setting from 
+/proc/meminfo: %s % e)
+
+
+def get_target_hugepages(self):
+
+Calculate the target number of hugepages for testing purposes.
+
+if self.vms  self.max_vms:
+self.vms = self.max_vms
+vmsm = (self.vms * self.mem) + (self.vms * 64)
+return int(vmsm * 1024 / self.hugepage_size)
+
+
+def set_hugepages(self):
+
+Sets the hugepage limit to the target hugepage value calculated.
+
+hugepage_cfg = open(/proc/sys/vm/nr_hugepages, r+)
+hp = hugepage_cfg.readline()
+while int(hp)  self.target_hugepages:
+loop_hp = hp
+hugepage_cfg.write(str(self.target_hugepages))
+hugepage_cfg.flush()
+time.sleep(5)
+hugepage_cfg.seek(0)
+hp = int(hugepage_cfg.readline())
+if loop_hp == hp:
+raise HugePageError(Cannot set the kernel hugepage setting 
+to the target value of %d hugepages. %
+self.target_hugepages)
+hugepage_cfg.close()
+
+
+def mount_hugepage_fs(self):
+
+