On Tue, Feb 9, 2016 at 11:51 AM, Paul Groeneweg | Pazion <p...@pazion.nl> wrote:
> For the record, my hosted-engine still runs el6. > You can run the 3.6 engine also on el6 so no issue there. > The hosts now run el7. > To reach 3.6 cluster compatibility level you need to upgrade all the hosts to el7. Now I don't understand where the engine VM is running. In order to be able to (re)deploy the host you upgraded to el7 you need a running engine. Is your engine VM running? If so, on which host? > This means I need to save backup and restore to a new instance? > I do need to upgrade hosted-engine to el7? > > I don't get the overall picture, but what I find strange is, the spUUID in > my hosted-engine.conf is not found. When I check the NFS hosted-engine > storageDomain it reports this same pool ID. Which in my opinion is the > reason for the failed ovf / vm.conf auto-import on shared storage is. > This is a bit tricky: when you setup hosted-engine on 3.4 or 3.5, the hosted-engine storage domain was on a dedicated (boostrap) storage pool that nobody was really using. In order to be able to manage and edit the engine VM from the engine, on 3.6 the hosted-engine storage domain should be attached to the storagePool of the datacenter as seen by the engine but to do that the hosted-engine storage domain should be detached from its bootstrap storage pool. The flow is this: 1. put everything into maintenance 2. upgrade ovirt-ha-agent to 3.6 3. restart ovirt-ha-agent: the new agent will - create a new volume with all the hosted-engine configuration on the shared volume - detach hosted-engine storage domain from its boostrap storage pool 4. when all the hosts are at 3.6, you can set 3.6 cluster compatibility level into the engine; this will trigger the autoimport of the hosted-engine storage domain from the engine and the hosted-engine storage domain will be attached to the datacenter storage pool. In your case is a bit different since your hosts was with el6 and so you cannot upgrade ovirt-ha-agent to 3.6. > > So to get this correct I need to backup and reinstall hosted-engine to get > the hosted engine working on my reinstalled hosts now runnign el7? > > Op di 9 feb. 2016 om 11:45 schreef Simone Tiraboschi <stira...@redhat.com > >: > >> On Tue, Feb 9, 2016 at 10:53 AM, Paul Groeneweg | Pazion <p...@pazion.nl> >> wrote: >> >>> The original (first host) was removed ( about a year a go I removed this >>> one ) and it is also removed from hosted-engine --vm-status overview. >>> >>> I ran 'hosted-engine --deploy' again on all the hosts because I >>> upgraded all hosts to EL7. >>> >> >> Upgrading hosted-engine from 3.5 on el6 to 3.6 on el7 requires some >> additional steps: >> https://bugzilla.redhat.com/show_bug.cgi?id=1188679 >> >> Adding Didi and Rafael who are working on that flow. >> >> >>> >>> ( I have als ran engine-setup for the upgrade ) >>> >>> How can I fix this missing spUUID issue? >>> Can I manually add this somehow or run somsetup again to fix this? >>> >>> >>> Op di 9 feb. 2016 om 10:43 schreef Simone Tiraboschi < >>> stira...@redhat.com>: >>> >>>> On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <p...@pazion.nl >>>> > wrote: >>>> >>>>> I have added 2 logs from 2 hosts >>>>> >>>>> >>>> Both the hosts are just additional host: >>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human >>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST >>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human >>>> dialog.__logString:219 DIALOG:SEND The specified storage >>>> location already contains a data domain. Is this an additional host setup >>>> (Yes, No)[Yes]? >>>> 2016-02-07 20:32:37 INFO >>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage >>>> storage._handleHostId:249 Installing on additional host >>>> >>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human >>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST >>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human >>>> dialog.__logString:219 DIALOG:SEND The specified storage >>>> location already contains a data domain. Is this an additional host setup >>>> (Yes, No)[Yes]? >>>> 2016-02-05 23:47:07 INFO >>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage >>>> storage._handleHostId:123 Installing on additional host >>>> >>>> Did you redeployed also the engine VM on a fresh storage domain or di >>>> you just run 'hosted-engine --deploy' again on all the hosts? >>>> If it's just an upgrade you don't have to run 'hosted-engine --deploy' >>>> again. >>>> >>>> Please follow this guide: >>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine >>>> >>>> >>>> >>>>> >>>>> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi < >>>>> stira...@redhat.com>: >>>>> >>>>>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion < >>>>>> p...@pazion.nl> wrote: >>>>>> >>>>>>> >>>>>>> I found vdsClient and ran a few commands. >>>>>>> >>>>>>> [root@geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo >>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>>>> >>>>>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>>>> >>>>>>> version = 3 >>>>>>> >>>>>>> role = Master >>>>>>> >>>>>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine >>>>>>> >>>>>>> type = NFS >>>>>>> >>>>>>> class = Data >>>>>>> >>>>>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4'] >>>>>>> >>>>>>> name = hostedengine_nfs >>>>>>> >>>>>>> >>>>>>> >>>>>> Can you please attach hosted-engine-setup logs? at the end of the >>>>>> setup procedure the hosted-engine setup shouldn't b attached to any >>>>>> storagePool. >>>>>> >>>>>> >>>>>>> [root@geisha-1 ~]# vdsClient -s 0 connectStoragePool >>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 >>>>>>> >>>>>>> Error using command: list index out of range >>>>>>> >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> >>>>>>> [root@geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo >>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 >>>>>>> >>>>>>> Unknown pool id, pool not connected: >>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',) >>>>>>> >>>>>>> >>>>>>> [root@geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList >>>>>>> >>>>>>> 00000002-0002-0002-0002-000000000385 >>>>>>> >>>>>>> >>>>>>> Can I manually create the missing storage pool which is missing with: >>>>>>> >>>>>>> vdsClient -s 0 createStoragePool >>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs >>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>>>> >>>>>>> If this would be the solution, I still wonder why there is no >>>>>>> existing storagePool for the storageDomain. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion < >>>>>>> p...@pazion.nl>: >>>>>>> >>>>>>>> I still keep getting >>>>>>>> >>>>>>>> The Hosted Engine Storage Domain doesn't exist. It should be >>>>>>>> imported into the setup. >>>>>>>> >>>>>>>> I went digging a little further and found this in the engine.log >>>>>>>> >>>>>>>> 2016-02-08 22:30:25,083 INFO >>>>>>>> >>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] >>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >>>>>>>> HSMGetStorageDomainInfoVDSCommand, return: >>>>>>>> <StorageDomainStatic:{name='hostedengine_nfs', >>>>>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'}, >>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427 >>>>>>>> 2016-02-08 22:30:25,083 INFO >>>>>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] >>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >>>>>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1 >>>>>>>> 2016-02-08 22:30:25,084 INFO >>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object >>>>>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}' >>>>>>>> 2016-02-08 22:30:25,102 WARN >>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action >>>>>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons: >>>>>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST >>>>>>>> >>>>>>>> So it looks like it can't find the storage domain. When I check >>>>>>>> vdsm.log I found an error: >>>>>>>> Thread-711902::ERROR::2016-02-08 >>>>>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': >>>>>>>> {'message': "Unknown pool id, pool not connected: >>>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}} >>>>>>>> >>>>>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is >>>>>>>> also trying to connect to in the engine.log. >>>>>>>> >>>>>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768 >>>>>>>> and https://bugzilla.redhat.com/show_bug.cgi?id=1294457 >>>>>>>> >>>>>>>> Can it be I have setup another DC with a master iscsi storage >>>>>>>> which is now offline? ( 1 DC with Cluster, VMs with hosted engine >>>>>>>> and >>>>>>>> master storage FC all up and running. ) >>>>>>>> >>>>>>>> Not sure where to go from here. Can I somehow restart this storage >>>>>>>> domain / do a new hosted-engine deploy with different settings? >>>>>>>> >>>>>>>> >>>>>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion < >>>>>>>> p...@pazion.nl>: >>>>>>>> >>>>>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi < >>>>>>>>> stira...@redhat.com>: >>>>>>>>> >>>>>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion < >>>>>>>>>> p...@pazion.nl> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for your response. >>>>>>>>>>> >>>>>>>>>>> I have reverted the commented entries and restarted >>>>>>>>>>> ovirt-ha-agent. >>>>>>>>>>> Now status on the daemon gives the error again: >>>>>>>>>>> http://screencast.com/t/DCZo6QJM1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment >>>>>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I have reinstalled host ( complete OS reinstall) and ran >>>>>>>>> hosted-engine --deploy for redeployment on an existing oVirt ( >>>>>>>>> hosted-engine ). >>>>>>>>> >>>>>>>>> As described in previous mail, during the deployment I got errors >>>>>>>>> about a missing hostedengine.metadata symlink to >>>>>>>>> /var/run/vdsd/storage/... >>>>>>>>> I created these by hand and then I ran deployment again and it >>>>>>>>> worked. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> The hosted engine domain is not shown in the web storage >>>>>>>>>>> overview... >>>>>>>>>>> When I start import it shows a warning: >>>>>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L >>>>>>>>>>> >>>>>>>>>>> Can I manually import the hostedengine storage? >>>>>>>>>>> ( Hosted engine is visible in web interface and I have running >>>>>>>>>>> VMs on another storage domain ). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It should auto-import it. >>>>>>>>>> Adding Roy here. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> I believe this auto-import is failing... >>>>>>>>> Where can I find logs about this auto-import? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi < >>>>>>>>>>> stira...@redhat.com>: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion < >>>>>>>>>>>> p...@pazion.nl> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Update: >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The OVF_STORE volume will contain the engine VM description in >>>>>>>>>>>> a format that the engine can easily handle. >>>>>>>>>>>> It will be created by the engine when: >>>>>>>>>>>> - it imported the hosted-engine storage domain >>>>>>>>>>>> - the engineVM is visible >>>>>>>>>>>> - the datacenter is up (you need at least one additional >>>>>>>>>>>> regular storage domain) >>>>>>>>>>>> >>>>>>>>>>>> Please make sure that all these conditions are met. >>>>>>>>>>>> >>>>>>>>>>>> If the OVF_STORE is not there or it's not accessible, >>>>>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from >>>>>>>>>>>> deployment time. >>>>>>>>>>>> You see a warning since we are failing back to an initial status >>>>>>>>>>>> and this >>>>>>>>>>>> can temporary override your latest customization to the engine VM >>>>>>>>>>>> configuration. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf >>>>>>>>>>>>> / vm.conf by commenting the conf_volume_UUID, conf_image_UUID >>>>>>>>>>>>> entries in >>>>>>>>>>>>> the hosted-engine.conf. After a restart of the ovirt-ha-agent the >>>>>>>>>>>>> errors >>>>>>>>>>>>> were gone. ( I believe this is not a permanent solution ). >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Please revert it. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2. This morning I noticed the warning in the event log. >>>>>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be >>>>>>>>>>>>> imported into the setup." >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It's just a side effect of your point 1, please revert it. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>>>> >>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>> >>>>>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion < >>>>>>>>>>>>> p...@pazion.nl>: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to >>>>>>>>>>>>>> run version 3.6. >>>>>>>>>>>>>> When deploying a new host have problems with the shared OVF >>>>>>>>>>>>>> storage config part. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. I get an error during deploy: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [ INFO ] Stage: Setup validation >>>>>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed >>>>>>>>>>>>>> to read metadata: [Errno 2] No such file or directory: >>>>>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata' >>>>>>>>>>>>>> [ INFO ] Stage: Clean up >>>>>>>>>>>>>> >>>>>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing. >>>>>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with >>>>>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks. >>>>>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went >>>>>>>>>>>>>> well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Now when the host is installed I notice the following in >>>>>>>>>>>>>> the logs >>>>>>>>>>>>>> >>>>>>>>>>>>>> .... >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing >>>>>>>>>>>>>> images >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading >>>>>>>>>>>>>> vm.conf from the shared storage domain >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying >>>>>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable >>>>>>>>>>>>>> to find >>>>>>>>>>>>>> OVF_STORE >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config >>>>>>>>>>>>>> ERROR Unable >>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable >>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to >>>>>>>>>>>>>> volume >>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in >>>>>>>>>>>>>> /rhev/data-center/mnt' - >>>>>>>>>>>>>> trying to restart agent >>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to >>>>>>>>>>>>>> volume >>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in >>>>>>>>>>>>>> /rhev/data-center/mnt' - >>>>>>>>>>>>>> trying to restart agent >>>>>>>>>>>>>> >>>>>>>>>>>>>> So at this point I was a bit stuck. >>>>>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for >>>>>>>>>>>>>> the hosted engine deploy. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This time all went without any problem directly. >>>>>>>>>>>>>> From there I retried upgrade and same problems again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found >>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314 >>>>>>>>>>>>>> I added no_subtree_check and tried deploy again, but again >>>>>>>>>>>>>> without luck. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I believe the hosted engine configs are not being migrated to >>>>>>>>>>>>>> the shared storage. And my problem has todo with >>>>>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage >>>>>>>>>>>>>> >>>>>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf, >>>>>>>>>>>>>> conf_volume_UUID, conf_image_UUID >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hosts run el7 >>>>>>>>>>>>>> Hosted engine runs el6 >>>>>>>>>>>>>> Hosted engine storage is through NFS3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web >>>>>>>>>>>>>> interface )? >>>>>>>>>>>>>> Or how can I create or trigger manually the OVF store the >>>>>>>>>>>>>> logs complain about? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>>> Pazion >>>>>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>>>>> >>>>>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>>>>> E p...@pazion.nl >>>>>>>>>>>>>> >>>>>>>>>>>>>> ***disclaimer*** >>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>>>>> information which is confidential and/or protected by >>>>>>>>>>>>>> intellectual property >>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) >>>>>>>>>>>>>> named above. >>>>>>>>>>>>>> Any use of the information contained herein (including, but not >>>>>>>>>>>>>> limited to, >>>>>>>>>>>>>> total or partial reproduction, communication or distribution in >>>>>>>>>>>>>> any form) >>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. >>>>>>>>>>>>>> If you >>>>>>>>>>>>>> have received this e-mail in error, please notify the sender >>>>>>>>>>>>>> either by >>>>>>>>>>>>>> telephone or by e-mail and delete the material from any >>>>>>>>>>>>>> computer. Thank you >>>>>>>>>>>>>> for your cooperation." >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>>>> >>>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>>> Pazion >>>>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>>>> >>>>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>>>> E p...@pazion.nl >>>>>>>>>>>>> >>>>>>>>>>>>> ***disclaimer*** >>>>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>>>> information which is confidential and/or protected by >>>>>>>>>>>>> intellectual property >>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) >>>>>>>>>>>>> named above. >>>>>>>>>>>>> Any use of the information contained herein (including, but not >>>>>>>>>>>>> limited to, >>>>>>>>>>>>> total or partial reproduction, communication or distribution in >>>>>>>>>>>>> any form) >>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. >>>>>>>>>>>>> If you >>>>>>>>>>>>> have received this e-mail in error, please notify the sender >>>>>>>>>>>>> either by >>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> for your cooperation." >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>> >>>>>>>>>>> Paul Groeneweg >>>>>>>>>>> Pazion >>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>> >>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>> E p...@pazion.nl >>>>>>>>>>> >>>>>>>>>>> ***disclaimer*** >>>>>>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>>>>>> which is confidential and/or protected by intellectual property >>>>>>>>>>> rights and >>>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any >>>>>>>>>>> use of >>>>>>>>>>> the information contained herein (including, but not limited to, >>>>>>>>>>> total or >>>>>>>>>>> partial reproduction, communication or distribution in any form) by >>>>>>>>>>> persons >>>>>>>>>>> other than the designated recipient(s) is prohibited. If you have >>>>>>>>>>> received >>>>>>>>>>> this e-mail in error, please notify the sender either by telephone >>>>>>>>>>> or by >>>>>>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>>>>>> cooperation." >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Met vriendelijke groeten, >>>>>>>>> >>>>>>>>> Paul Groeneweg >>>>>>>>> Pazion >>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>> >>>>>>>>> T +31 26 3020038 >>>>>>>>> M +31 614 277 577 >>>>>>>>> E p...@pazion.nl >>>>>>>>> >>>>>>>>> ***disclaimer*** >>>>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>>>> which is confidential and/or protected by intellectual property >>>>>>>>> rights and >>>>>>>>> are intended for the sole use of the recipient(s) named above. Any >>>>>>>>> use of >>>>>>>>> the information contained herein (including, but not limited to, >>>>>>>>> total or >>>>>>>>> partial reproduction, communication or distribution in any form) by >>>>>>>>> persons >>>>>>>>> other than the designated recipient(s) is prohibited. If you have >>>>>>>>> received >>>>>>>>> this e-mail in error, please notify the sender either by telephone or >>>>>>>>> by >>>>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>>>> cooperation." >>>>>>>>> >>>>>>>> -- >>>>>>>> Met vriendelijke groeten, >>>>>>>> >>>>>>>> Paul Groeneweg >>>>>>>> Pazion >>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>> >>>>>>>> T +31 26 3020038 >>>>>>>> M +31 614 277 577 >>>>>>>> E p...@pazion.nl >>>>>>>> >>>>>>>> ***disclaimer*** >>>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>>> which is confidential and/or protected by intellectual property rights >>>>>>>> and >>>>>>>> are intended for the sole use of the recipient(s) named above. Any use >>>>>>>> of >>>>>>>> the information contained herein (including, but not limited to, total >>>>>>>> or >>>>>>>> partial reproduction, communication or distribution in any form) by >>>>>>>> persons >>>>>>>> other than the designated recipient(s) is prohibited. If you have >>>>>>>> received >>>>>>>> this e-mail in error, please notify the sender either by telephone or >>>>>>>> by >>>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>>> cooperation." >>>>>>>> >>>>>>> -- >>>>>>> Met vriendelijke groeten, >>>>>>> >>>>>>> Paul Groeneweg >>>>>>> Pazion >>>>>>> Webdevelopment - Hosting - Apps >>>>>>> >>>>>>> T +31 26 3020038 >>>>>>> M +31 614 277 577 >>>>>>> E p...@pazion.nl >>>>>>> >>>>>>> ***disclaimer*** >>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>> which is confidential and/or protected by intellectual property rights >>>>>>> and >>>>>>> are intended for the sole use of the recipient(s) named above. Any use >>>>>>> of >>>>>>> the information contained herein (including, but not limited to, total >>>>>>> or >>>>>>> partial reproduction, communication or distribution in any form) by >>>>>>> persons >>>>>>> other than the designated recipient(s) is prohibited. If you have >>>>>>> received >>>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>> cooperation." >>>>>>> >>>>>> -- >>>>> Met vriendelijke groeten, >>>>> >>>>> Paul Groeneweg >>>>> Pazion >>>>> Webdevelopment - Hosting - Apps >>>>> >>>>> T +31 26 3020038 >>>>> M +31 614 277 577 >>>>> E p...@pazion.nl >>>>> >>>>> ***disclaimer*** >>>>> "This e-mail and any attachments thereto may contain information which >>>>> is confidential and/or protected by intellectual property rights and are >>>>> intended for the sole use of the recipient(s) named above. Any use of the >>>>> information contained herein (including, but not limited to, total or >>>>> partial reproduction, communication or distribution in any form) by >>>>> persons >>>>> other than the designated recipient(s) is prohibited. If you have received >>>>> this e-mail in error, please notify the sender either by telephone or by >>>>> e-mail and delete the material from any computer. Thank you for your >>>>> cooperation." >>>>> >>>> -- >>> Met vriendelijke groeten, >>> >>> Paul Groeneweg >>> Pazion >>> Webdevelopment - Hosting - Apps >>> >>> T +31 26 3020038 >>> M +31 614 277 577 >>> E p...@pazion.nl >>> >>> ***disclaimer*** >>> "This e-mail and any attachments thereto may contain information which >>> is confidential and/or protected by intellectual property rights and are >>> intended for the sole use of the recipient(s) named above. Any use of the >>> information contained herein (including, but not limited to, total or >>> partial reproduction, communication or distribution in any form) by persons >>> other than the designated recipient(s) is prohibited. If you have received >>> this e-mail in error, please notify the sender either by telephone or by >>> e-mail and delete the material from any computer. Thank you for your >>> cooperation." >>> >> -- > Met vriendelijke groeten, > > Paul Groeneweg > Pazion > Webdevelopment - Hosting - Apps > > T +31 26 3020038 > M +31 614 277 577 > E p...@pazion.nl > > ***disclaimer*** > "This e-mail and any attachments thereto may contain information which is > confidential and/or protected by intellectual property rights and are > intended for the sole use of the recipient(s) named above. Any use of the > information contained herein (including, but not limited to, total or > partial reproduction, communication or distribution in any form) by persons > other than the designated recipient(s) is prohibited. If you have received > this e-mail in error, please notify the sender either by telephone or by > e-mail and delete the material from any computer. Thank you for your > cooperation." >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
