Re: [ovirt-devel] [VDSM] Tests failing because of ordering dependencies

2016-10-12 Thread Dan Kenigsberg 
On Wed, Oct 12, 2016 at 09:16:08PM +0300, Nir Soffer wrote:
> If you want to reproduce the errors, try this patch:
> https://gerrit.ovirt.org/65404
> 
> On Wed, Oct 12, 2016 at 9:14 PM, Nir Soffer  wrote:
> > Hi all,
> >
> > Trying to run vdsm tests via tox (so correct nose is used automatically),
> > some of the tests fail.
> >
> > The failure are all about ordering expectations, which look wrong.
> >
> > Please check and fix your tests.
> >
> > Thanks,
> > Nir
> >
> > 
> >
> > 18:04:10 
> > ==
> > 18:04:10 FAIL: test_parseVolumeStatus (gluster_cli_tests.GlusterCliTests)
> > 18:04:10 
> > --
> > 18:04:10 Traceback (most recent call last):
> > 18:04:10   File
> > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
> > line 1121, in test_parseVolumeStatus
> > 18:04:10 self._parseVolumeStatusClients_test()
> > 18:04:10   File
> > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
> > line 449, in _parseVolumeStatusClients_test
> > 18:04:10 self.assertEquals(status.keys(), ['bricks', 'name'])
> >
> > status.keys() order is not undefined.

I've never seen this in Python 2, but since Python 3 uses a different
hash function, this specific issue is handled in
https://gerrit.ovirt.org/#/c/65008/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] engine-setup: ***L:ERROR Internal error: No module named dwh

2016-10-12 Thread Simone Tiraboschi 
Hi,
Jakub was right: the issue has been introduced with commit 221c7ed.

It affects just the dev env since there we are currently not installing (we
need to dig this!) DWH which instead is a dependency on the packaged builds
since 4.0.

Patch https://gerrit.ovirt.org/#/c/65409/ fixes it.

ciao

On Wed, Oct 12, 2016 at 6:08 PM, Vojtech Szocs  wrote:

> Same problem here.
>
> Vojtech
>
>
> - Original Message -
> > From: "Dominik Holler" 
> > To: devel@ovirt.org
> > Sent: Wednesday, October 12, 2016 2:39:27 PM
> > Subject: Re: [ovirt-devel] engine-setup: ***L:ERROR Internal error: No
> module named dwh
> >
> > Hi,
> > I got the same error. Is there any experience about this, by now?
> > Dominik
> >
> > On Mon, 10 Oct 2016 15:48:08 -0400 (EDT)
> > Jakub Niedermertl  wrote:
> >
> > > Hi all,
> > >
> > > does anyone else experience following error of `engine-setup`?
> > >
> > > $ ~/target/bin/engine-setup
> > > ***L:ERROR Internal error: No module named dwh
> > >
> > > I have a suspicion it might be related to commit '221c7ed packaging:
> > > setup: Remove constants duplication'.
> > >
> > > Jakub
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> >
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> >
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Can't add DC with API v4 - client issue

2016-10-12 Thread Yaniv Kaul 
On Fri, Oct 7, 2016 at 10:44 PM, Yaniv Kaul  wrote:

> I'm trying on FC24, using python-ovirt-engine-sdk4-4.1.0
> -0.0.20161003git056315d.fc24.x86_64 to add a DC, and failing - against
> master. The client is unhappy:
> File 
> "/home/ykaul/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py",
> line 98, in add_dc4
> version=sdk4.types.Version(major=DC_VER_MAJ,minor=DC_VER_MIN),
>   File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line
> 4347, in add
> response = self._connection.send(request)
>   File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
> 276, in send
> return self.__send(request)
>   File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
> 298, in __send
> self._sso_token = self._get_access_token()
>   File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
> 460, in _get_access_token
> sso_response = self._get_sso_response(self._sso_url, post_data)
>   File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line
> 498, in _get_sso_response
> return json.loads(body_buf.getvalue().decode('utf-8'))
>   File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
> return _default_decoder.decode(s)
>   File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>   File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
>
> Surprisingly, I now can't find that RPM of this SDK in resources.ovirt.org
>  now.
>
> I've tried with http://resources.ovirt.org/pub/ovirt-master-snapshot/rp
> m/fc24/x86_64/python-ovirt-engine-sdk4-4.0.0-0.1.20161004git
> f94eeb5.fc24.x86_64.rpm
>
> - same result.
>
> Did not see anything obvious on server or engine logs.
> The code:
> def add_dc4(api):
> nt.assert_true(api != None)
> dcs_service = api.system_service().data_centers_service()
> nt.assert_true(
> dc = dcs_service.add(
> sdk4.types.DataCenter(
> name=DC_NAME4,
> description='APIv4 DC',
> local=False,
> version=sdk4.types.Version(maj
> or=DC_VER_MAJ,minor=DC_VER_MIN),
> ),
> )
> )
>
>
> And the api object is from:
> return sdk4.Connection(
> url=url,
> username=constants.ENGINE_USER,
> password=str(self.metadata['ovirt-engine-password']),
> insecure=True,
> debug=True,
> )
>
>
The clue is actually on the HTTPd logs:
192.168.203.1 - - [12/Oct/2016:17:56:27 -0400] "POST
/ovirt-engine/sso/oauth/token HTTP/1.1" 404 74

And indeed, from the deubg log:
begin captured logging << \n
root: DEBUG: Trying 192.168.203.3...\n
root: DEBUG: Connected to 192.168.203.3 (192.168.203.3) port 443 (#0)\n
root: DEBUG: Initializing NSS with certpath: sql:/etc/pki/nssdb\n
root: DEBUG: skipping SSL peer certificate verification\n
root: DEBUG: ALPN/NPN, server did not agree to a protocol\n
root: DEBUG: SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\n
root: DEBUG: Server certificate:\n
root: DEBUG: subject: CN=engine,O=Test,C=US\n
root: DEBUG: start date: Oct 11 21:55:29 2016 GMT\n
root: DEBUG: expire date: Sep 16 21:55:29 2021 GMT\n
root: DEBUG: common name: engine\nroot: DEBUG: issuer:
CN=engine.38998,O=Test,C=US\n
*root: DEBUG: POST /ovirt-engine/sso/oauth/token HTTP/1.1\n*
*root: DEBUG: Host: 192.168.203.3\n*
*root: DEBUG: User-Agent: PythonSDK/4.1.0a0\n*
*root: DEBUG: Accept: application/json\n*
*root: DEBUG: Content-Length: 78\n*
*root: DEBUG: Content-Type: application/x-www-form-urlencoded\nroot: DEBUG:
username=admin%40internal=ovirt-app-api=123_type=password\n*
*root: DEBUG: upload completely sent off: 78 out of 78 bytes\n*
*root: DEBUG: HTTP/1.1 404 Not Found\n*
*root: DEBUG: Date: Wed, 12 Oct 2016 21:56:27 GMT\n*
*root: DEBUG: Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips\n*
*root: DEBUG: Content-Length: 74\n*
*root: DEBUG: Content-Type: text/html; charset=UTF-8\n*
*root: DEBUG: \n*
*root: DEBUG: Error404 - Not
Found\n*
root: DEBUG: Connection #0 to host 192.168.203.3 left intact\n
- >> end captured logging
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Tests failing because of ordering dependencies

2016-10-12 Thread Nir Soffer 
On Wed, Oct 12, 2016 at 9:14 PM, Nir Soffer  wrote:
> Trying to run vdsm tests via tox (so correct nose is used automatically),
> some of the tests fail.
>
> The failure are all about ordering expectations, which look wrong.

tox -e tests
tests create: /vdsm/.tox/tests
tests installdeps: nose==1.3.7
tests installed: nose==1.3.7
tests runtests: PYTHONHASHSEED='1382456816'

tox uses random seed on each run, exposing bad code assuming order
of unordered things.

To repeat tests failures, you probably want to use the same seed,
this can be done like this:

tox -e tests --hashseed SEED

See https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHASHSEED

Nir
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Tests failing because of ordering dependencies

2016-10-12 Thread Nir Soffer 
On Wed, Oct 12, 2016 at 9:14 PM, Nir Soffer  wrote:
> 18:04:10 self.assertEquals(status.keys(), ['bricks', 'name'])
>
> status.keys() order is not undefined.

*is* undefined

(I was interrupted by a child while writing this)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [VDSM] Tests failing because of ordering dependencies

2016-10-12 Thread Nir Soffer 
If you want to reproduce the errors, try this patch:
https://gerrit.ovirt.org/65404

On Wed, Oct 12, 2016 at 9:14 PM, Nir Soffer  wrote:
> Hi all,
>
> Trying to run vdsm tests via tox (so correct nose is used automatically),
> some of the tests fail.
>
> The failure are all about ordering expectations, which look wrong.
>
> Please check and fix your tests.
>
> Thanks,
> Nir
>
> 
>
> 18:04:10 
> ==
> 18:04:10 FAIL: test_parseVolumeStatus (gluster_cli_tests.GlusterCliTests)
> 18:04:10 
> --
> 18:04:10 Traceback (most recent call last):
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
> line 1121, in test_parseVolumeStatus
> 18:04:10 self._parseVolumeStatusClients_test()
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
> line 449, in _parseVolumeStatusClients_test
> 18:04:10 self.assertEquals(status.keys(), ['bricks', 'name'])
>
> status.keys() order is not undefined.
>
> 18:04:10 AssertionError: Lists differ: ['name', 'bricks'] != ['bricks', 
> 'name']
> 18:04:10
> 18:04:10 First differing element 0:
> 18:04:10 name
> 18:04:10 bricks
> 18:04:10
> 18:04:10 - ['name', 'bricks']
> 18:04:10 + ['bricks', 'name']
> 18:04:10
> 18:04:10 
> ==
> 18:04:10 FAIL: testSetPolicyParameters (momTests.MomPolicyTests)
> 18:04:10 
> --
> 18:04:10 Traceback (most recent call last):
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/momTests.py",
> line 118, in testSetPolicyParameters
> 18:04:10 self.assertEqual(api.last_policy_content, expected)
> 18:04:10 AssertionError: "(set a 5)\n(set b True)\n(set c 'test')" !=
> "(set a 5)\n(set c 'test')\n(set b True)"
>
> Nothing obvious, need deeper checking.
>
> 18:04:10  >> begin captured logging << 
> 
> 18:04:10 2016-10-12 18:01:56,902 INFO(MainThread) [MOM] Preparing
> MOM interface (momIF:49)
> 18:04:10 2016-10-12 18:01:56,903 INFO(MainThread) [MOM] Using
> named unix socket /tmp/tmpqOQZvm/test_mom_vdsm.sock (momIF:58)
> 18:04:10 - >> end captured logging << 
> -
> 18:04:10
> 18:04:10 
> ==
> 18:04:10 FAIL: test_disk_virtio_cache (vmStorageTests.DriveXMLTests)
> 18:04:10 
> --
> 18:04:10 Traceback (most recent call last):
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmStorageTests.py",
> line 84, in test_disk_virtio_cache
> 18:04:10 self.check(vm_conf, conf, xml, is_block_device=False)
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmStorageTests.py",
> line 222, in check
> 18:04:10 self.assertXMLEqual(drive.getXML().toxml(), xml)
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py",
> line 253, in assertXMLEqual
> 18:04:10 (actualXML, expectedXML))
> 18:04:10 AssertionError: XMLs are different:
> 18:04:10 Actual:
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10 54-a672-23e5b495a9ea
> 18:04:10  io="threads" name="qemu" type="qcow2" />
> 18:04:10 
> 18:04:10 800
> 18:04:10 612
> 18:04:10 
> 18:04:10 
> 18:04:10
> 18:04:10 Expected:
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10 54-a672-23e5b495a9ea
> 18:04:10  io="threads" name="qemu" type="qcow2" />
> 18:04:10 
> 18:04:10 612
> 18:04:10 800
>
> Order of these elements differ, need to check why.
>
> 18:04:10 
> 18:04:10 
> 18:04:10
> 18:04:10
> 18:04:10
> 18:04:10 
> ==
> 18:04:10 FAIL: testCpuXML (vmTests.TestVm)
> 18:04:10 
> --
> 18:04:10 Traceback (most recent call last):
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmTests.py",
> line 434, in testCpuXML
> 18:04:10 self.assertXMLEqual(find_xml_element(xml, "./cputune"), 
> cputuneXML)
> 18:04:10   File
> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py",
> line 253, in assertXMLEqual
> 18:04:10 (actualXML, expectedXML))
> 18:04:10 AssertionError: XMLs are different:
> 18:04:10 Actual:
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10 
> 18:04:10
> 18:04:10 Expected:
> 18:04:10 
> 18:04:10 
> 18:04:10 
>
> Same
>
> 18:04:10 
> 18:04:10
> 18:04:10
> 18:04:10
> 18:04:10 
>

[ovirt-devel] [VDSM] Tests failing because of ordering dependencies

2016-10-12 Thread Nir Soffer 
Hi all,

Trying to run vdsm tests via tox (so correct nose is used automatically),
some of the tests fail.

The failure are all about ordering expectations, which look wrong.

Please check and fix your tests.

Thanks,
Nir



18:04:10 ==
18:04:10 FAIL: test_parseVolumeStatus (gluster_cli_tests.GlusterCliTests)
18:04:10 --
18:04:10 Traceback (most recent call last):
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
line 1121, in test_parseVolumeStatus
18:04:10 self._parseVolumeStatusClients_test()
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/gluster_cli_tests.py",
line 449, in _parseVolumeStatusClients_test
18:04:10 self.assertEquals(status.keys(), ['bricks', 'name'])

status.keys() order is not undefined.

18:04:10 AssertionError: Lists differ: ['name', 'bricks'] != ['bricks', 'name']
18:04:10
18:04:10 First differing element 0:
18:04:10 name
18:04:10 bricks
18:04:10
18:04:10 - ['name', 'bricks']
18:04:10 + ['bricks', 'name']
18:04:10
18:04:10 ==
18:04:10 FAIL: testSetPolicyParameters (momTests.MomPolicyTests)
18:04:10 --
18:04:10 Traceback (most recent call last):
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/momTests.py",
line 118, in testSetPolicyParameters
18:04:10 self.assertEqual(api.last_policy_content, expected)
18:04:10 AssertionError: "(set a 5)\n(set b True)\n(set c 'test')" !=
"(set a 5)\n(set c 'test')\n(set b True)"

Nothing obvious, need deeper checking.

18:04:10  >> begin captured logging << 
18:04:10 2016-10-12 18:01:56,902 INFO(MainThread) [MOM] Preparing
MOM interface (momIF:49)
18:04:10 2016-10-12 18:01:56,903 INFO(MainThread) [MOM] Using
named unix socket /tmp/tmpqOQZvm/test_mom_vdsm.sock (momIF:58)
18:04:10 - >> end captured logging << -
18:04:10
18:04:10 ==
18:04:10 FAIL: test_disk_virtio_cache (vmStorageTests.DriveXMLTests)
18:04:10 --
18:04:10 Traceback (most recent call last):
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmStorageTests.py",
line 84, in test_disk_virtio_cache
18:04:10 self.check(vm_conf, conf, xml, is_block_device=False)
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmStorageTests.py",
line 222, in check
18:04:10 self.assertXMLEqual(drive.getXML().toxml(), xml)
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py",
line 253, in assertXMLEqual
18:04:10 (actualXML, expectedXML))
18:04:10 AssertionError: XMLs are different:
18:04:10 Actual:
18:04:10 
18:04:10 
18:04:10 
18:04:10 
18:04:10 54-a672-23e5b495a9ea
18:04:10 
18:04:10 
18:04:10 800
18:04:10 612
18:04:10 
18:04:10 
18:04:10
18:04:10 Expected:
18:04:10 
18:04:10 
18:04:10 
18:04:10 
18:04:10 54-a672-23e5b495a9ea
18:04:10 
18:04:10 
18:04:10 612
18:04:10 800

Order of these elements differ, need to check why.

18:04:10 
18:04:10 
18:04:10
18:04:10
18:04:10
18:04:10 ==
18:04:10 FAIL: testCpuXML (vmTests.TestVm)
18:04:10 --
18:04:10 Traceback (most recent call last):
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmTests.py",
line 434, in testCpuXML
18:04:10 self.assertXMLEqual(find_xml_element(xml, "./cputune"), cputuneXML)
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/testlib.py",
line 253, in assertXMLEqual
18:04:10 (actualXML, expectedXML))
18:04:10 AssertionError: XMLs are different:
18:04:10 Actual:
18:04:10 
18:04:10 
18:04:10 
18:04:10 
18:04:10
18:04:10 Expected:
18:04:10 
18:04:10 
18:04:10 

Same

18:04:10 
18:04:10
18:04:10
18:04:10
18:04:10 ==
18:04:10 FAIL: testSetIoTune (vmTests.TestVm)
18:04:10 --
18:04:10 Traceback (most recent call last):
18:04:10   File
"/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/vmTests.py",
line 936, in testSetIoTune
18:04:10 self._xml_sanitizer(expected_xml))
18:04:10 AssertionError: '102'
!= '210'
18:04:10  >> begin captured logging << 
18:04:10 2016-10-12

Re: [ovirt-devel] engine-setup: ***L:ERROR Internal error: No module named dwh

2016-10-12 Thread Vojtech Szocs 
Same problem here.

Vojtech


- Original Message -
> From: "Dominik Holler" 
> To: devel@ovirt.org
> Sent: Wednesday, October 12, 2016 2:39:27 PM
> Subject: Re: [ovirt-devel] engine-setup: ***L:ERROR Internal error: No module 
> named dwh
> 
> Hi,
> I got the same error. Is there any experience about this, by now?
> Dominik
> 
> On Mon, 10 Oct 2016 15:48:08 -0400 (EDT)
> Jakub Niedermertl  wrote:
> 
> > Hi all,
> > 
> > does anyone else experience following error of `engine-setup`?
> > 
> > $ ~/target/bin/engine-setup
> > ***L:ERROR Internal error: No module named dwh
> > 
> > I have a suspicion it might be related to commit '221c7ed packaging:
> > setup: Remove constants duplication'.
> > 
> > Jakub
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> 
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
> 
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] engine-setup: ***L:ERROR Internal error: No module named dwh

2016-10-12 Thread Sandro Bonazzola 
Adding some relevant people.

On Wed, Oct 12, 2016 at 2:39 PM, Dominik Holler  wrote:

> Hi,
> I got the same error. Is there any experience about this, by now?
> Dominik
>
> On Mon, 10 Oct 2016 15:48:08 -0400 (EDT)
> Jakub Niedermertl  wrote:
>
> > Hi all,
> >
> > does anyone else experience following error of `engine-setup`?
> >
> > $ ~/target/bin/engine-setup
> > ***L:ERROR Internal error: No module named dwh
> >
> > I have a suspicion it might be related to commit '221c7ed packaging:
> > setup: Remove constants duplication'.
> >
> > Jakub
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>



-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] [ACTION REQUIRED] oVirt 4.0.5 RC2 build starting

2016-10-12 Thread Sandro Bonazzola 
Fyi oVirt products maintainers,
An oVirt build for an official release is going to start right now.
If you're a maintainer for any of the projects included in oVirt
distribution and you have changes in your package ready to be released
please:
- bump version and release to be GA ready
- tag your release within git (implies a GitHub Release to be automatically
created)
- build your packages within jenkins / koji / copr / whatever
- verify all bugs on MODIFIED have target release and target milestone set.
- add your builds to releng-tools/releases/ovirt-4.0.5_rc2.conf within
releng-tools project

-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] exploring a possible integration between collectd and Vdsm

2016-10-12 Thread Francesco Romani 
- Original Message -
> From: "Francesco Romani" 
> To: "devel" 
> Sent: Wednesday, October 12, 2016 9:29:37 AM
> Subject: Re: [ovirt-devel] [vdsm] exploring a possible integration between 
> collectd and Vdsm
> 
> - Original Message -
> > From: "Yaniv Kaul" 
> > To: "Francesco Romani" 
> > Cc: "devel" 
> > Sent: Tuesday, October 11, 2016 10:31:14 PM
> > Subject: Re: [ovirt-devel] [vdsm] exploring a possible integration between
> > collectd and Vdsm
> > 
> > On Tue, Oct 11, 2016 at 2:05 PM, Francesco Romani 
> > wrote:
> > 
> > > Hi all,
> > >
> > > In the last 2.5 days I was exploring if and how we can integrate collectd
> > > and Vdsm.
> [...]
> > This generally sounds like a good idea - and I hope it is coordinated with
> > our efforts for monitoring (see [1], [2]).
> 
> Sure it will. 

Sure it will be coordinated with the monitoring efforts*.

-- 
Francesco Romani
Red Hat Engineering Virtualization R & D
Phone: 8261328
IRC: fromani
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [vdsm] exploring a possible integration between collectd and Vdsm

2016-10-12 Thread Francesco Romani 
- Original Message -
> From: "Yaniv Kaul" 
> To: "Francesco Romani" 
> Cc: "devel" 
> Sent: Tuesday, October 11, 2016 10:31:14 PM
> Subject: Re: [ovirt-devel] [vdsm] exploring a possible integration between 
> collectd and Vdsm
> 
> On Tue, Oct 11, 2016 at 2:05 PM, Francesco Romani 
> wrote:
> 
> > Hi all,
> >
> > In the last 2.5 days I was exploring if and how we can integrate collectd
> > and Vdsm.
[...]
> This generally sounds like a good idea - and I hope it is coordinated with
> our efforts for monitoring (see [1], [2]).

Sure it will. I played a couple of days with collectd to "just" see
a. how hard is to write a collectd plugin, and/or if it is feasible to
   ship ot out-of-tree for the initial few releases, until it stabilizes
   so it can be submitted upstream
b. if we can get events/notifications from collectd
c. if we can integrate those notifications with Vdsm

And turns out we *can* do all of the above, with various degrees of difficulty.

> Few notes:
> - I think the most compelling reason to move to collectd is actually to
> benefit from the already existing plugins that it already has, which will
> cover
> a lot of the missing monitoring requirements and wishes we have (example:
> local disk usage on the host), as well as integrate it into
> Engine monitoring (example: postgresql performance monitoring).

Agreed

> - You can't remove monitoring from VDSM - as it new VDSM may work against
> older Engine setups. You can gradually remove them.

Yes, for example we can make Vdsm poll collectd and act as facade to old 
Engines,
while new one should skip this step and ask collectd or the metrics aggregator
service you mention below.

> I'd actually begin with cleanup - there are some 'metrics' that are simply
> not needed and should not be reported in the first place and
> are there for historic reasons only. Remove them - from Engine first, from
> the DB and all, then later we can either send fake values or remove
> from VDSM.

Yes, this is the first place where we need to coordinate with the metrics 
effort.

> - If you are moving to collectd, as you can see from the metrics effort,
> we'd really want to send it elsewhere - and Engine should consume it from
> there.
> Metrics storages usually have a very nice REST UI with the ability to bring
> series with average, with different criteria (such as per hour, per minute
> or what not stats), etc.

Fully agreed

> - I agree with Nir about separating between our core business and the
> monitoring we do for extra. Keep in mind that some of the stats are for SLA
> and critical scheduling decisions as well.

Yes, of course adding a dependency for core monitoring is risky.
So far the bottom line is that relying on collectd for this is just one more
option on the table now.

[mostly brainstorming from now on]

However, I'd like highlight that is not just risky: is a different tradeoff.
Doing the core monitoring in Vdsm (so in python, essentially in a single 
threaded server)
is not a free lunch, because this has a quite high price on performance level.

If the main Vdsm process is overloaded, then the polling cycle can get longer, 
and the
overall response time of processing system events (e.g. disk detected full) can 
get
longer as well.
We've observed in not-so-distant past high response time from heavily loaded 
Vdsm.

I think the idea of having different instances for different monitoring purposes
(credit to Nir) is the best shot at the moment.
We could maybe have one standard system collectd for regular monitoring,
and perhaps one special purpose, very limited collectd instance for critical 
information.
On top of that, Vdsm could double-checl and keep doing the core monitoring 
itself,
albeit at lower rate (e.g. every 10s instead of every 2s; every 60s instead of 
every 15s).

Leveraging libvirt events is *the* right thing, no doubt about that, but it 
would be very nice
to have a dependable external service which can generate the events we need 
based on
libvirt data, and move the notification logic on top of it.

Something like (final picture, excluding intermediate compatibility layers)

[data source]
-+---
 |
 `-> [monitoring/metrics collection]
  -+---
   |
   +--> [metrics store] -{data}-> [Engine]
   |
   `--> [notification service] -{events}-> [Vdsm]


Not all the "boxes" need to be separate processes, for example collectd has some
support for thresholds and notifications which is ~80% of what Vdsm needs 
(again not
considering reliability, just feature-wise).  

[end brainstorm]

> - The libvirt collectd plugin is REALLY outdated. I think it may require
> significant work to bring it up to speed with our existing capabilities.

Yep I looked briefly at that code. It is REALLY outdated :)
Besides some information just not reported