I am working on providing correlation id between the engine and vdsm. The engine patch which bumps jsonrpc version (already merged) is still open. It contains update [1] for the engine.
I thought that once we base our artifacts using version (tag) we are not using head. I need to have a way to introduce new features. [1] https://gerrit.ovirt.org/#/c/75333/2/backend/manager/dependencies/common/src/main/modules/org/ovirt/vdsm-jsonrpc-java/main/module.xml On Wed, Apr 12, 2017 at 9:03 AM, Michal Skrivanek < [email protected]> wrote: > > On 12 Apr 2017, at 08:37, Yaniv Kaul <[email protected]> wrote: > > > > On Wed, Apr 12, 2017 at 9:24 AM, Barak Korren <[email protected]> wrote: > >> This seems to have been the Secodary SD issue I reported on Sunday: >> http://lists.ovirt.org/pipermail/devel/2017-April/030139.html >> >> That was up until build #6307 where AddHost started failing: >> http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6307 >> > > That's an entirely different game: > *2017-04-11 11:24:36,783-04 ERROR > [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] > (org.ovirt.thread.pool-7-thread-1) [3497aa05] Error: > java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils* > 2017-04-11 11:24:36,783-04 ERROR [org.ovirt.engine.core. > vdsbroker.vdsbroker.PollVDSCommand] (org.ovirt.thread.pool-7-thread-1) > [3497aa05] Exception: java.util.concurrent.ExecutionException: > java.lang.NoClassDefFoundError: > > > > this one started in http://jenkins.ovirt.org/job/test-repo_ovirt_ > experimental_master/6307/ > > there are 2 patches linkded, one is in vdsm-jsonrpc-java, which is what > seems to be mismatched > Piotr/Sandro, are all artifacts published correctly? > > Thanks, > michal > > org/apache/commons/lang/StringUtils > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > [rt.jar:1.8.0_121] > at java.util.concurrent.FutureTask.get(FutureTask.java:206) > [rt.jar:1.8.0_121] > at > org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:69) > [vdsbroker.jar:] > at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.getValue( > HostSetupNetworkPoller.java:56) [bll.jar:] > at org.ovirt.engine.core.bll.network.host.HostSetupNetworkPoller.poll( > HostSetupNetworkPoller.java:41) [bll.jar:] > > ... > Caused by: java.lang.NoClassDefFoundError: org/apache/commons/lang/ > StringUtils > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message. > withCorrelationId(Message.java:75) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.SSLStompClient. > sendMessage(SSLStompClient.java:85) [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:83) > [vdsm-jsonrpc-java-client.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.FutureMap.<init>(FutureMap.java:91) > [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.lambda$ > timeBoundPoll$1(JsonRpcVdsServer.java:965) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ > FutureCallable.call(JsonRpcVdsServer.java:484) [vdsbroker.jar:] > at org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer$ > FutureCallable.call(JsonRpcVdsServer.java:473) [vdsbroker.jar:] > > > >> >> >> >> On 11 April 2017 at 18:12, Piotr Kliczewski <[email protected]> >> wrote: >> >>> I looks like SD activate failed on the engine side and the spm >>> connection was gently closed by the engine: >>> >>> 2017-04-11 09:10:24,807-04 INFO [org.ovirt.engine.core.bll.st >>> orage.domain.ActivateStorageDomainCommand] (default task-29) [2b66a42b] >>> ActivateStorage Domain. After Connect all hosts to pool. Time: Tue Apr 11 >>> 09:10:24 EDT 2017 >>> 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto >>> rage.pool.RefreshPoolSingleAsyncOperation] >>> (org.ovirt.thread.pool-7-thread-29) [2b66a42b] Could not connect vds >>> 'lago-basic-suite-master-host0' to pool 'test-dc' - moving host to >>> non-operational: null >>> 2017-04-11 09:10:24,842-04 ERROR [org.ovirt.engine.core.bll.sto >>> rage.pool.RefreshPoolSingleAsyncOperation] >>> (org.ovirt.thread.pool-7-thread-28) [2b66a42b] Could not connect vds >>> 'lago-basic-suite-master-host1' to pool 'test-dc' - moving host to >>> non-operational: null >>> 2017-04-11 09:10:24,904-04 INFO [org.ovirt.engine.core.bll.Se >>> <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] >>> (default task-29) [2bc83b1c] Running command: SetNonOperationalVdsCommand >>> internal: true. Entities affected : ID: >>> e1b611f2-67fb-4acb-8261-07133134e0e7 >>> Type: VDS >>> 2017-04-11 09:10:24,909-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (default task-29) [2bc83b1c] START, SetVdsStatusVDSCommand(HostName = >>> lago-basic-suite-master-host0, >>> SetVdsStatusVDSCommandParameters:{runAsync='true', >>> hostId='e1b611f2-67fb-4acb-8261-07133134e0e7', status='NonOperational', >>> nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', >>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 5c45239 >>> 2017-04-11 09:10:24,919-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (default task-29) [2bc83b1c] FINISH, SetVdsStatusVDSCommand, log id: 5c45239 >>> 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb >>> roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) >>> [2bc83b1c] Compiled stored procedure. Call string is [{call >>> getvmsrunningonvds(?)}] >>> 2017-04-11 09:10:24,922-04 DEBUG [org.ovirt.engine.core.dal.dbb >>> roker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-29) >>> [2bc83b1c] SqlCall for procedure [GetVmsRunningOnVds] compiled >>> 2017-04-11 09:10:24,985-04 WARN [org.ovirt.engine.core.dal.db >>> broker.auditloghandling.AuditLogDirector] (default task-29) [2bc83b1c] >>> EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 2bc83b1c, >>> Job ID: d6455a45-48ad-42cd-9815-e9826944b9a0, Call Stack: null, Custom >>> Event ID: -1, Message: Host lago-basic-suite-master-host0 cannot access the >>> Storage Domain(s) iscsi attached to the Data Center test-dc. Setting Host >>> state to Non-Operational. >>> 2017-04-11 09:10:24,985-04 DEBUG >>> [org.ovirt.engine.core.utils.timer.FixedDelayJobListener] >>> (DefaultQuartzScheduler4) [] Rescheduling DEFAULT.org.ovirt.engine.core. >>> vdsbroker.VdsManager.onTimer#-9223372036854775741 as there is no >>> unfired trigger. >>> 2017-04-11 09:10:25,036-04 INFO [org.ovirt.engine.core.bll.Se >>> <http://org.ovirt.engine.core.bll.se/>tNonOperationalVdsCommand] >>> (default task-29) [2ffa5981] Running command: SetNonOperationalVdsCommand >>> internal: true. Entities affected : ID: >>> 87a5f691-ea5c-4692-a4ab-52dad0dafe34 >>> Type: VDS >>> 2017-04-11 09:10:25,039-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (default task-29) [2ffa5981] START, SetVdsStatusVDSCommand(HostName = >>> lago-basic-suite-master-host1, >>> SetVdsStatusVDSCommandParameters:{runAsync='true', >>> hostId='87a5f691-ea5c-4692-a4ab-52dad0dafe34', status='NonOperational', >>> nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', >>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 706ffc3b >>> >>> ..... >>> >>> 2017-04-11 09:10:25,366-04 INFO [org.ovirt.engine.core.vdsbro >>> ker.vdsbroker.SpmStopVDSCommand] (default task-29) [2ffa5981] FINISH, >>> SpmStopVDSCommand, log id: 62f82cfb >>> 2017-04-11 09:10:25,366-04 DEBUG >>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] >>> (default task-29) [2ffa5981] UNSUBSCRIBE >>> id:4cd59c67-1b57-4456-9a16-52a72ac6f689 >>> >>> \00 >>> 2017-04-11 09:10:25,367-04 DEBUG [org.ovirt.vdsm.jsonrpc.client >>> .reactors.stomp.StompCommonClient] (default task-29) [2ffa5981] Message >>> sent: UNSUBSCRIBE >>> >>> >>> >>> On Tue, Apr 11, 2017 at 4:49 PM, Sandro Bonazzola <[email protected]> >>> wrote: >>> >>>> Link to Job: http://jenkins.ovirt.org/job/t >>>> est-repo_ovirt_experimental_master/6303 >>>> >>>> >>>> *OST is failing in the last 50 builds* >>>> >>>> >>>> >>>> *00:18:22.090* [basic_suit_el7] # add_secondary_storage_domains: >>>> *00:18:22.090* [basic_suit_el7] Error while running thread*00:18:22.090* >>>> [basic_suit_el7] Traceback (most recent call last):*00:18:22.090* >>>> [basic_suit_el7] File "/usr/lib/python2.7/site-packages/lago/utils.py", >>>> line 57, in _ret_via_queue*00:18:22.090* [basic_suit_el7] >>>> queue.put({'return': func()})*00:18:22.090* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 427, in add_nfs_storage_domain*00:18:22.090* [basic_suit_el7] >>>> add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, >>>> SD_NFS_PATH, nfs_version='v4_2')*00:18:22.091* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 441, in add_generic_nfs_storage_domain*00:18:22.091* >>>> [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, >>>> nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.091* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 490, in add_generic_nfs_storage_domain_4*00:18:22.091* >>>> [basic_suit_el7] host=_random_host_from_dc_4(api, >>>> DC_NAME),*00:18:22.091* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 106, in _random_host_from_dc_4*00:18:22.091* [basic_suit_el7] >>>> return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.091* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 100, in _hosts_in_dc_4*00:18:22.091* [basic_suit_el7] raise >>>> RuntimeError('Could not find hosts that are up in DC %s' % >>>> dc_name)*00:18:22.092* [basic_suit_el7] RuntimeError: Could not find hosts >>>> that are up in DC test-dc*00:18:22.092* [basic_suit_el7] Error while >>>> running thread*00:18:22.092* [basic_suit_el7] Traceback (most recent call >>>> last):*00:18:22.092* [basic_suit_el7] File >>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in >>>> _ret_via_queue*00:18:22.092* [basic_suit_el7] queue.put({'return': >>>> func()})*00:18:22.092* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 583, in add_iso_storage_domain*00:18:22.092* [basic_suit_el7] >>>> add_generic_nfs_storage_domain(prefix, SD_ISO_NAME, SD_ISO_HOST_NAME, >>>> SD_ISO_PATH, sd_format='v1', sd_type='iso', >>>> nfs_version='v3')*00:18:22.092* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 441, in add_generic_nfs_storage_domain*00:18:22.092* >>>> [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, >>>> nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.093* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 490, in add_generic_nfs_storage_domain_4*00:18:22.093* >>>> [basic_suit_el7] host=_random_host_from_dc_4(api, >>>> DC_NAME),*00:18:22.093* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 106, in _random_host_from_dc_4*00:18:22.093* [basic_suit_el7] >>>> return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.093* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 100, in _hosts_in_dc_4*00:18:22.093* [basic_suit_el7] raise >>>> RuntimeError('Could not find hosts that are up in DC %s' % >>>> dc_name)*00:18:22.093* [basic_suit_el7] RuntimeError: Could not find hosts >>>> that are up in DC test-dc*00:18:22.094* [basic_suit_el7] Error while >>>> running thread*00:18:22.094* [basic_suit_el7] Traceback (most recent call >>>> last):*00:18:22.094* [basic_suit_el7] File >>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in >>>> _ret_via_queue*00:18:22.094* [basic_suit_el7] queue.put({'return': >>>> func()})*00:18:22.094* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 587, in add_templates_storage_domain*00:18:22.094* [basic_suit_el7] >>>> add_generic_nfs_storage_domain(prefix, SD_TEMPLATES_NAME, >>>> SD_TEMPLATES_HOST_NAME, SD_TEMPLATES_PATH, sd_format='v1', >>>> sd_type='export', nfs_version='v4_1')*00:18:22.094* [basic_suit_el7] >>>> File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 441, in add_generic_nfs_storage_domain*00:18:22.094* >>>> [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, >>>> nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.094* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 490, in add_generic_nfs_storage_domain_4*00:18:22.095* >>>> [basic_suit_el7] host=_random_host_from_dc_4(api, >>>> DC_NAME),*00:18:22.095* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 106, in _random_host_from_dc_4*00:18:22.095* [basic_suit_el7] >>>> return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.095* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 100, in _hosts_in_dc_4*00:18:22.095* [basic_suit_el7] raise >>>> RuntimeError('Could not find hosts that are up in DC %s' % >>>> dc_name)*00:18:22.095* [basic_suit_el7] RuntimeError: Could not find hosts >>>> that are up in DC test-dc*00:18:22.095* [basic_suit_el7] Error while >>>> running thread*00:18:22.095* [basic_suit_el7] Traceback (most recent call >>>> last):*00:18:22.096* [basic_suit_el7] File >>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in >>>> _ret_via_queue*00:18:22.096* [basic_suit_el7] queue.put({'return': >>>> func()})*00:18:22.096* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 436, in add_second_nfs_storage_domain*00:18:22.096* [basic_suit_el7] >>>> SD_NFS_HOST_NAME, SD_SECOND_NFS_PATH)*00:18:22.096* [basic_suit_el7] >>>> File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 441, in add_generic_nfs_storage_domain*00:18:22.096* >>>> [basic_suit_el7] add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, >>>> nfs_host_name, mount_path, sd_format, sd_type, nfs_version)*00:18:22.096* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 490, in add_generic_nfs_storage_domain_4*00:18:22.096* >>>> [basic_suit_el7] host=_random_host_from_dc_4(api, >>>> DC_NAME),*00:18:22.096* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 106, in _random_host_from_dc_4*00:18:22.097* [basic_suit_el7] >>>> return random.choice(_hosts_in_dc_4(api, dc_name))*00:18:22.097* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 100, in _hosts_in_dc_4*00:18:22.097* [basic_suit_el7] raise >>>> RuntimeError('Could not find hosts that are up in DC %s' % >>>> dc_name)*00:18:22.097* [basic_suit_el7] RuntimeError: Could not find hosts >>>> that are up in DC test-dc*00:18:22.097* [basic_suit_el7] Error while >>>> running thread*00:18:22.097* [basic_suit_el7] Traceback (most recent call >>>> last):*00:18:22.097* [basic_suit_el7] File >>>> "/usr/lib/python2.7/site-packages/lago/utils.py", line 57, in >>>> _ret_via_queue*00:18:22.098* [basic_suit_el7] queue.put({'return': >>>> func()})*00:18:22.098* [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 803, in import_template_from_glance*00:18:22.098* [basic_suit_el7] >>>> generic_import_from_glance(api, image_name=CIRROS_IMAGE_NAME, >>>> image_ext='_glance_template', as_template=True)*00:18:22.098* >>>> [basic_suit_el7] File >>>> "/home/jenkins/workspace/test-repo_ovirt_experimental_master/ovirt-system-tests/basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 637, in generic_import_from_glance*00:18:22.098* [basic_suit_el7] >>>> target_image.import_image(import_action)*00:18:22.098* [basic_suit_el7] >>>> File >>>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", >>>> line 26017, in import_image*00:18:22.098* [basic_suit_el7] >>>> headers={"Correlation-Id":correlation_id}*00:18:22.098* [basic_suit_el7] >>>> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", >>>> line 122, in request*00:18:22.098* [basic_suit_el7] >>>> persistent_auth=self.__persistent_auth*00:18:22.099* [basic_suit_el7] >>>> File >>>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", >>>> line 79, in do_request*00:18:22.099* [basic_suit_el7] >>>> persistent_auth)*00:18:22.099* [basic_suit_el7] File >>>> "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", >>>> line 162, in __do_request*00:18:22.099* [basic_suit_el7] raise >>>> errors.RequestError(response_code, response_reason, >>>> response_body)*00:18:22.099* [basic_suit_el7] RequestError: *00:18:22.099* >>>> [basic_suit_el7] status: 400*00:18:22.099* [basic_suit_el7] reason: Bad >>>> Request*00:18:22.099* [basic_suit_el7] detail: Cannot import Virtual Disk: >>>> Storage Domain cannot be accessed.*00:18:22.099* [basic_suit_el7] -Please >>>> check that at least one Host is operational and Data Center state is up. >>>> >>>> >>>> -- >>>> SANDRO BONAZZOLA >>>> >>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> >>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> [email protected] >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> [email protected] >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >> >> -- >> Barak Korren >> [email protected] >> RHCE, RHCi, RHV-DevOps Team >> https://ifireball.wordpress.com/ >> >> _______________________________________________ >> Devel mailing list >> [email protected] >> http://lists.ovirt.org/mailman/listinfo/devel >> > > _______________________________________________ > Devel mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/devel > > > > _______________________________________________ > Devel mailing list > [email protected] > http://lists.ovirt.org/mailman/listinfo/devel >
_______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
