Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> can I know a use case for this 'live copy metadata or ' the 'only way > to access device tags when hot-attach? my thought is this is one time > thing in cloud-init side either through metatdata service or config > drive and won't be used later? then why I need a live copy? If I do something like this: nova interface-attach --tag=data-network --port-id=foo myserver Then we update the device metadata live, which is visible immediately via the metadata service. However, in config drive, that only gets updated the next time the drive is generated (which may be a long time away). For more information on device metadata, see: https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html Further, some of the drivers support setting the admin password securely via metadata, which similarly requires the instance pulling updated information out, which wouldn't be available in the config drive. For reference: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1985-L1993 --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
we are continue to evaluating the ways to remove the restrictions in the
future, one question on following comments:
>>>Why don't you support the metadata service? That's a pretty fundamental
mechanism for nova and openstack. It's the only way you can get a live
copy of metadata, and it's the only way you can get access to device
tags when you hot-attach something. Personally, I think that it's
something that needs to work.
As Matt mentioned in https://review.openstack.org/#/c/562154/ PS#4
As far as I know the metadata service is not a basic feature, it's optional
and some deployments don't run it because of possible security concerns.
so seems it's different suggestion,... and for the following suggestion
It's the only way you can get a live copy of metadata, and it's the only
way you can get access to device tags when you hot-attach something
can I know a use case for this 'live copy metadata or ' the 'only way to
access device tags when hot-attach?
my thought is this is one time thing in cloud-init side either through
metatdata service or config drive and won't be used later? then why I need
a live copy?
and because nova do the hot attach why it's the only way to access the
tags? what exec in the deployed VM will access the device? cloud-init or
something else?
Thanks a lot for your help
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Dan Smith
To: "Chen CH Ji"
Cc: "OpenStack Development Mailing List \(not for usage questions
\)"
Date: 04/13/2018 09:46 PM
Subject:Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
> for the run_validation=False issue, you are right, because z/VM driver
> only support config drive and don't support metadata service ,we made
> bad assumption and took wrong action to disabled the whole ssh check,
> actually according to [1] , we should only disable
> CONF.compute_feature_enabled.metadata_service but keep both
> self.run_ssh and CONF.compute_feature_enabled.config_drive as True in
> order to make config drive test validation take effect, our CI will
> handle that
Why don't you support the metadata service? That's a pretty fundamental
mechanism for nova and openstack. It's the only way you can get a live
copy of metadata, and it's the only way you can get access to device
tags when you hot-attach something. Personally, I think that it's
something that needs to work.
> For the tgz/iso9660 question below, this is because we got wrong info
> from low layer component folks back to 2012 and after discuss with
> some experts again, actually we can create iso9660 in the driver layer
> and pass down to the spawned virtual machine and during startup
> process, the VM itself will mount the iso file and consume it, because
> from linux perspective, either tgz or iso9660 doesn't matter , only
> need some files in order to transfer the information from openstack
> compute node to the spawned VM. so our action is to change the format
> from tgz to iso9660 and keep consistent to other drivers.
The "iso file" will not be inside the guest, but rather passed to the
guest as a block device, right?
> For the config drive working mechanism question, according to [2] z/VM
> is Type 1 hypervisor while Qemu/KVM are mostly likely to be Type 2
> hypervisor, there is no file system in z/VM hypervisor (I omit too
> much detail here) , so we can't do something like linux operation
> system to keep a file as qcow2 image in the host operating system,
I'm not sure what the type-1-ness has to do with this. The hypervisor
doesn't need to support any specific filesystem for this to work. Many
drivers we have in the tree are type-1 (xen, vmware, hyperv, powervm)
and you can argue that KVM is type-1-ish. They support configdrive.
> what we do is use a special file pool to store the config drive and
> during VM init process, we read that file from special device and
> attach to VM as iso9660 format then cloud-init will handle the follow
> up, the cloud-init handle process is identical to other platform
This and the previous mention of this sort of behavior has me
concerned. Are you describing some sort of process that runs when the
instance is starting to initialize its environment, or something that
runs *inside* the instance and thus functionality that has to exist in
the *image* to work?
--Dan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Thanks for sharing this info , it make sense to leave a -2 here, I will
keep modifying follow on patches and get more reviews. thanks
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Dan Smith
To: "Chen CH Ji"
Cc: "OpenStack Development Mailing List \(not for usage questions
\)"
Date: 04/30/2018 10:55 PM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
> According to requirements and comments, now we opened the CI runs with
> run_validation = True And according to [1] below, for example, [2]
> need the ssh validation passed the test
>
> And there are a couple of comments need some enhancement on the logs
> of CI such as format and legacy incorrect links of logs etc the newest
> logs sample can be found [3] (take n-cpu as example and those logs are
> with _white.html)
>
> Also, the blueprint [4] requested by previous discussion post here
> again for reference
>
> please let us know whether the procedure -2 can be removed in order to
> proceed . thanks for your help
The CI log format issues look fixed to me and validation is turned on
for the stuff supported, which is what was keeping it out of the
runway.
I still plan to leave the -2 on there until the next few patches have
agreement, just so we don't land an empty shell driver before we are
sure we're going to land spawn/destroy, etc. That's pretty normal
procedure and I'll be around to remove it when appropriate.
--Dan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Mon, 30 Apr 2018 12:42:40 -0400, Matthew Treinish wrote: On Mon, Apr 30, 2018 at 09:21:22AM -0700, melanie witt wrote: On Fri, 27 Apr 2018 17:40:20 +0800, Chen Ch Ji wrote: According to requirements and comments, now we opened the CI runs with run_validation = True And according to [1] below, for example, [2] need the ssh validation passed the test And there are a couple of comments need some enhancement on the logs of CI such as format and legacy incorrect links of logs etc the newest logs sample can be found [3] (take n-cpu as example and those logs are with _white.html) Also, the blueprint [4] requested by previous discussion post here again for reference Thank you for alerting us about the completion of the work on the z/VM CI. The logs look much improved and ssh connectivity and metadata functionality via config drive is being verified by tempest. The only strange thing I noticed is it appears tempest starts multiple times in the log [0]. Do you know what's going on there? This is normal, it's an artifact of a few things. The first time config is dumped to the logs is because of tempest verify-config being run as part of devstack: https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L590 You also see the API requests this command is making being logged. Then when the tempest tests are actually being run the config is dumped to the logs once per test worker process. Basically every time we parse the config file at debug log levels it get's printed to the log file. FWIW, you can also see this in a gate run too: http://logs.openstack.org/90/539590/10/gate/tempest-full/4b0a136/controller/logs/tempest_log.txt A-ha, thanks for sharing all of that info. I have learned something new. :) -melanie __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Mon, Apr 30, 2018 at 09:21:22AM -0700, melanie witt wrote: > On Fri, 27 Apr 2018 17:40:20 +0800, Chen Ch Ji wrote: > > According to requirements and comments, now we opened the CI runs with > > run_validation = True > > And according to [1] below, for example, [2] need the ssh validation > > passed the test > > > > And there are a couple of comments need some enhancement on the logs of > > CI such as format and legacy incorrect links of logs etc > > the newest logs sample can be found [3] (take n-cpu as example and those > > logs are with _white.html) > > > > Also, the blueprint [4] requested by previous discussion post here again > > for reference > > Thank you for alerting us about the completion of the work on the z/VM CI. > The logs look much improved and ssh connectivity and metadata functionality > via config drive is being verified by tempest. > > The only strange thing I noticed is it appears tempest starts multiple times > in the log [0]. Do you know what's going on there? This is normal, it's an artifact of a few things. The first time config is dumped to the logs is because of tempest verify-config being run as part of devstack: https://github.com/openstack-dev/devstack/blob/master/lib/tempest#L590 You also see the API requests this command is making being logged. Then when the tempest tests are actually being run the config is dumped to the logs once per test worker process. Basically every time we parse the config file at debug log levels it get's printed to the log file. FWIW, you can also see this in a gate run too: http://logs.openstack.org/90/539590/10/gate/tempest-full/4b0a136/controller/logs/tempest_log.txt -Matt Treinish > > That said, since things are looking good with z/VM CI now, we've added the > z/VM patch series back into a review runway today. > > Cheers, > -melanie > > [0] > http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-17444/logs/tempest.log > from https://review.openstack.org/527658 > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Fri, 27 Apr 2018 17:40:20 +0800, Chen Ch Ji wrote: According to requirements and comments, now we opened the CI runs with run_validation = True And according to [1] below, for example, [2] need the ssh validation passed the test And there are a couple of comments need some enhancement on the logs of CI such as format and legacy incorrect links of logs etc the newest logs sample can be found [3] (take n-cpu as example and those logs are with _white.html) Also, the blueprint [4] requested by previous discussion post here again for reference Thank you for alerting us about the completion of the work on the z/VM CI. The logs look much improved and ssh connectivity and metadata functionality via config drive is being verified by tempest. The only strange thing I noticed is it appears tempest starts multiple times in the log [0]. Do you know what's going on there? That said, since things are looking good with z/VM CI now, we've added the z/VM patch series back into a review runway today. Cheers, -melanie [0] http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-17444/logs/tempest.log from https://review.openstack.org/527658 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> According to requirements and comments, now we opened the CI runs with > run_validation = True And according to [1] below, for example, [2] > need the ssh validation passed the test > > And there are a couple of comments need some enhancement on the logs > of CI such as format and legacy incorrect links of logs etc the newest > logs sample can be found [3] (take n-cpu as example and those logs are > with _white.html) > > Also, the blueprint [4] requested by previous discussion post here > again for reference > > please let us know whether the procedure -2 can be removed in order to > proceed . thanks for your help The CI log format issues look fixed to me and validation is turned on for the stuff supported, which is what was keeping it out of the runway. I still plan to leave the -2 on there until the next few patches have agreement, just so we don't land an empty shell driver before we are sure we're going to land spawn/destroy, etc. That's pretty normal procedure and I'll be around to remove it when appropriate. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
According to requirements and comments, now we opened the CI runs with
run_validation = True
And according to [1] below, for example, [2] need the ssh validation passed
the test
And there are a couple of comments need some enhancement on the logs of CI
such as format and legacy incorrect links of logs etc
the newest logs sample can be found [3] (take n-cpu as example and those
logs are with _white.html)
Also, the blueprint [4] requested by previous discussion post here again
for reference
please let us know whether the procedure -2 can be removed in order to
proceed . thanks for your help
[1]
http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-17455/logs/tempest.log
2018-04-27 08:50:44.852 19582 DEBUG tempest [-] validation.run_validation
= True
http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-17455/console.html
{0}
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
[86.788179s] ... ok
[2]
https://github.com/openstack/tempest/blob/master/tempest/scenario/test_server_basic_ops.py
[3]
http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-17455/logs/n-cpu.log_white.html
[4] https://review.openstack.org/#/c/562154/
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: melanie witt
To: openstack-dev@lists.openstack.org
Date: 04/18/2018 01:41 AM
Subject:Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
On Tue, 17 Apr 2018 16:58:22 +0800, Chen Ch Ji wrote:
> For the question on AE documentation, it's open source in [1] and the
> documentation for how to build and use is [2]
> once our code is upstream, there are a set of documentation change which
> will cover this image build process by
> adding some links to there [3]
Thanks, that is good info.
> You are right, we need image to have our Active Engine, I think
> different arch and platform might have their unique
> requirements and our solution our Active Engine is very like to
> cloud-init so no harm to add it from user's perspective
> I think later we can upload image to some place so anyone is able to
> consume it as test image if they like
> because different arch's image (e.g x86 and s390x) can't be shared
anyway.
>
> For the config drive format you mentioned, actually, as previous
> explanation and discussion witho Michael and Dan,
> We found the iso9660 can be used (previously we made a bad assumption)
> and we already changed the patch in [4],
> so it's exactly same to other virt drivers you mentioned , we don't need
> special format and iso9660 works perfect for our driver
That's good news, I'm glad that got resolved.
> It make sense to me we are temply moved out from runway, I suppose we
> can adjust the CI to enable the run_ssh = true
> with config drive functionalities very soon and we will apply for review
> after that with the test result requested in our CI log.
Okay, sounds good. Since you expect to be up and running with
[validation]run_validation = True soon, I'm going to move the z/VM
driver blueprint back to the front of the queue and put the next
blueprint in line into the runway.
Then, when the next blueprint end date arrives (currently 2018-04-30),
if the z/VM CI is ready with cleaned up, human readable log files and is
running with run_ssh = True with the test_server_basic_ops test to
verify config drive operation, we will add the z/VM driver blueprint
back to a runway for dedicated review.
Let us know when the z/VM CI is ready, in case other runway reviews are
completed early. If other runway reviews complete early, a runway space
might be available earlier than 2018-04-30.
Thanks,
-melanie
> Thanks
>
> [1]
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mfcloud_python-2Dzvm-2Dsdk_blob_master_tools_share_zvmguestconfigure&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=CRsbKwMgE5rCqLl8MTo6ZnKA4QkxK3NRmDont5BYcqw&s=RpjRNK6wiUJDNTYKBkou6nSDpaUkNOXdmBJ-SyjkPaw&e=
> [2]
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__cloudlib4zvm.readthedocs.io_en_latest_makeimage.html-23configuration-2Dof-2Dactivation-2Dengine-2Dae-2Din-2Dzlinux&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=CRsbKwMgE5rCqLl8MTo6ZnKA4QkxK3NRmDont5BYcqw&s=CVvkU6HtWW7GArGIpFT4fichM0fuTXXrmWD9zyRo9h0&e=
> [3]
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__review.openstack.org_-23_q_status-3Aopen-2Bproject-3Aopenstack_nova-2Bbranch-3Amaster-2Btopic-3Abp_add-2Dzvm-2Ddriver-2Drocky&
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Yes, fully understand this ,thanks for sharing!
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Jim Rollenhagen
To: "OpenStack Development Mailing List (not for usage questions)"
Date: 04/20/2018 10:13 PM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
On Fri, Apr 20, 2018 at 4:02 AM, Chen CH Ji wrote:
Thanks a lot for your sharing, that's good info, just curious why [1]
need zip and base64 encode if my understand is correct
I was told nova need format should be pure vfat or iso9660, I assume it's
because actually the config drive itself is making by iso by default
then wrap a zip/base64 format ... thanks
We only gzip and base64 to send it to the ironic API. It is decoded and
unzipped before writing it to disk, so it is a pure iso9660 on the disk.
// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=4L-KwemnBkUdTMyGA_BviipEqJ7MKNGlKFMKH6J6iaM&s=S52V2lLNK1Mh7rprSl-edF3Q2M4m3qEXcWd3jTW8Y9g&e=
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Fri, Apr 20, 2018 at 4:02 AM, Chen CH Ji wrote: > Thanks a lot for your sharing, that's good info, just curious why [1] need > zip and base64 encode if my understand is correct > I was told nova need format should be pure vfat or iso9660, I assume it's > because actually the config drive itself is making by iso by default > then wrap a zip/base64 format ... thanks > We only gzip and base64 to send it to the ironic API. It is decoded and unzipped before writing it to disk, so it is a pure iso9660 on the disk. // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Thanks a lot for your sharing, that's good info, just curious why [1] need
zip and base64 encode if my understand is correct
I was told nova need format should be pure vfat or iso9660, I assume it's
because actually the config drive itself is making by iso by default
then wrap a zip/base64 format ... thanks
[1]
https://github.com/openstack/nova/blob/324899c621ee02d877122ba3412712ebb92831f2/nova/virt/ironic/driver.py#L977
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Jim Rollenhagen
To: "OpenStack Development Mailing List (not for usage questions)"
Date: 04/19/2018 12:02 AM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
On Wed, Apr 18, 2018 at 10:56 AM, Matthew Booth wrote:
> I agree with Mikal that needing more agent behavior than cloud-init
does
> a disservice to the users.
>
> I feel like we get a lot of "but no, my hypervisor is special!"
> reasoning when people go to add a driver to nova. So far, I think
> they're a lot more similar than people think. Ironic is the weirdest
one
> we have (IMHO and no offense to the ironic folks) and it can support
> configdrive properly.
I was going to ask this. Even if the contents of the disk can't be
transferred in advance... how does ironic do this? There must be a
way.
I'm not sure if this is a rhetorical question, so I'll just answer it. :)
We basically build the configdrive in nova-compute, then gzip and base64
it, and send it to ironic with the deploy request. On the ironic side, we
unpack it and write it to the end of the boot disk.
https://github.com/openstack/nova/blob/324899c621ee02d877122ba3412712ebb92831f2/nova/virt/ironic/driver.py#L952-L985
// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=oGbSbjsg1aTX1kg1lAQIkEY8abfQ9632w8YmP3Lrt-U&s=Q8_XusVUibDx7ee5WguroOVm00Fl4rw2XSNEHIVOGb0&e=
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> Having briefly read the cloud-init snippet which was linked earlier in > this thread, the requirement seems to be that the guest exposes the > device as /dev/srX or /dev/cdX. So I guess in order to make this work: > > * You need to tell z/VM to expose the virtual disk as an optical disk > * The z/VM kernel needs to call optical disks /dev/srX or /dev/cdX According to the docs, it doesn't need to be. You can indicate the configdrive via filesystem label which makes sense given we support vfat for it as well. http://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html#version-2 --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Wed, Apr 18, 2018 at 10:56 AM, Matthew Booth wrote: > > > I agree with Mikal that needing more agent behavior than cloud-init does > > a disservice to the users. > > > > I feel like we get a lot of "but no, my hypervisor is special!" > > reasoning when people go to add a driver to nova. So far, I think > > they're a lot more similar than people think. Ironic is the weirdest one > > we have (IMHO and no offense to the ironic folks) and it can support > > configdrive properly. > > I was going to ask this. Even if the contents of the disk can't be > transferred in advance... how does ironic do this? There must be a > way. > I'm not sure if this is a rhetorical question, so I'll just answer it. :) We basically build the configdrive in nova-compute, then gzip and base64 it, and send it to ironic with the deploy request. On the ironic side, we unpack it and write it to the end of the boot disk. https://github.com/openstack/nova/blob/324899c621ee02d877122ba3412712ebb92831f2/nova/virt/ironic/driver.py#L952-L985 // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On 18 April 2018 at 15:10, Dan Smith wrote: >> Thanks for the concern and fully under it , the major reason is >> cloud-init doesn't have a hook or plugin before it start to read >> config drive (ISO disk) z/VM is an old hypervisor and no way to do >> something like libvirt to define a ISO format disk in xml definition, >> instead, it can define disks in the defintion of virtual machine and >> let VM to decide its format. >> >> so we need a way to tell cloud-init where to find ISO file before >> cloud-init start but without AE, we can't handle that...some update on >> the spec here for further information >> https://review.openstack.org/#/c/562154/ > > The ISO format does not come from telling libvirt something about > it. The host creates and formats the image, adds the data, and then > attaches it to the instance. The latter part is the only step that > involves configuring libvirt to attach the image to the instance. The > rest is just stuff done by nova-compute (and the virt driver) on the > linux system it's running on. That's the same arrangement as your > driver, AFAICT. > > You're asking the system to hypervisor (or something running on it) to > grab the image from glance, pre-filled with data. This is no different, > except that the configdrive image comes from the system running the > compute service. I don't see how it's any different in actual hypervisor > mechanics, and thus feel like there _has_ to be a way to do this without > the AE magic agent. Having briefly read the cloud-init snippet which was linked earlier in this thread, the requirement seems to be that the guest exposes the device as /dev/srX or /dev/cdX. So I guess in order to make this work: * You need to tell z/VM to expose the virtual disk as an optical disk * The z/VM kernel needs to call optical disks /dev/srX or /dev/cdX > I agree with Mikal that needing more agent behavior than cloud-init does > a disservice to the users. > > I feel like we get a lot of "but no, my hypervisor is special!" > reasoning when people go to add a driver to nova. So far, I think > they're a lot more similar than people think. Ironic is the weirdest one > we have (IMHO and no offense to the ironic folks) and it can support > configdrive properly. I was going to ask this. Even if the contents of the disk can't be transferred in advance... how does ironic do this? There must be a way. Matt -- Matthew Booth Red Hat OpenStack Engineer, Compute DFG Phone: +442070094448 (UK) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> Thanks for the concern and fully under it , the major reason is > cloud-init doesn't have a hook or plugin before it start to read > config drive (ISO disk) z/VM is an old hypervisor and no way to do > something like libvirt to define a ISO format disk in xml definition, > instead, it can define disks in the defintion of virtual machine and > let VM to decide its format. > > so we need a way to tell cloud-init where to find ISO file before > cloud-init start but without AE, we can't handle that...some update on > the spec here for further information > https://review.openstack.org/#/c/562154/ The ISO format does not come from telling libvirt something about it. The host creates and formats the image, adds the data, and then attaches it to the instance. The latter part is the only step that involves configuring libvirt to attach the image to the instance. The rest is just stuff done by nova-compute (and the virt driver) on the linux system it's running on. That's the same arrangement as your driver, AFAICT. You're asking the system to hypervisor (or something running on it) to grab the image from glance, pre-filled with data. This is no different, except that the configdrive image comes from the system running the compute service. I don't see how it's any different in actual hypervisor mechanics, and thus feel like there _has_ to be a way to do this without the AE magic agent. I agree with Mikal that needing more agent behavior than cloud-init does a disservice to the users. I feel like we get a lot of "but no, my hypervisor is special!" reasoning when people go to add a driver to nova. So far, I think they're a lot more similar than people think. Ironic is the weirdest one we have (IMHO and no offense to the ironic folks) and it can support configdrive properly. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Thanks for the concern and fully under it , the major reason is cloud-init
doesn't have a hook or plugin before it start to read config drive (ISO
disk)
z/VM is an old hypervisor and no way to do something like libvirt to define
a ISO format disk in xml definition, instead, it can define disks in the
defintion
of virtual machine and let VM to decide its format.
so we need a way to tell cloud-init where to find ISO file before
cloud-init start but without AE, we can't handle that...some update on the
spec here
for further information https://review.openstack.org/#/c/562154/
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Michael Still
To: "OpenStack Development Mailing List (not for usage questions)"
Date: 04/18/2018 05:08 PM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
I'm confused about the design of AE to be honest. Is there a good reason
that this functionality couldn't be provided by cloud-init? I think there's
a lot of cost in deviating from the industry standard, so the reasons to do
so have to be really solid.
I'm also a bit confused by what seems to be support for streaming
configuration. Is there any documentation on the design of AE anywhere?
Thanks,
Michael
On Tue, Apr 17, 2018 at 6:58 PM, Chen CH Ji wrote:
For the question on AE documentation, it's open source in [1] and the
documentation for how to build and use is [2]
once our code is upstream, there are a set of documentation change which
will cover this image build process by
adding some links to there [3]
You are right, we need image to have our Active Engine, I think different
arch and platform might have their unique
requirements and our solution our Active Engine is very like to
cloud-init so no harm to add it from user's perspective
I think later we can upload image to some place so anyone is able to
consume it as test image if they like
because different arch's image (e.g x86 and s390x) can't be shared
anyway.
For the config drive format you mentioned, actually, as previous
explanation and discussion witho Michael and Dan,
We found the iso9660 can be used (previously we made a bad assumption)
and we already changed the patch in [4],
so it's exactly same to other virt drivers you mentioned , we don't need
special format and iso9660 works perfect for our driver
It make sense to me we are temply moved out from runway, I suppose we can
adjust the CI to enable the run_ssh = true
with config drive functionalities very soon and we will apply for review
after that with the test result requested in our CI log.
Thanks
[1]
https://github.com/mfcloud/python-zvm-sdk/blob/master/tools/share/zvmguestconfigure
[2]
http://cloudlib4zvm.readthedocs.io/en/latest/makeimage.html#configuration-of-activation-engine-ae-in-zlinux
[3] https://review.openstack.org/#/q/status:open+project:openstack/nova
+branch:master+topic:bp/add-zvm-driver-rocky
[4] https://review.openstack.org/#/c/527658/33/nova/virt/zvm/utils.py
line 104
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
Inactive hide details for melanie witt ---04/17/2018 09:21:03 AM---On
Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote: > >>>melanie witt
---04/17/2018 09:21:03 AM---On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch
Ji wrote: > >>>The "iso file" will not be inside the gu
From: melanie witt
To: openstack-dev@lists.openstack.org
Date: 04/17/2018 09:21 AM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote:
> >>>The "iso file" will not be inside the guest, but rather passed to
> the guest as a block device, right?
> Cloud init expects to find a config drive with following requirements
> [1], in order to make cloud init able to consume config drive , we
> should be able to prepare it,
> in some hypervisor, you can define something like following to the VM
> then VM startup is able to consume it
>
> but for z/VM case it allows disk to be created during VM create (define
> )stage but no disk format set, it's the operating system's
> responsibility to define the purpose of the
> disk, so what we do is
> 1) first when we build image ,we create a small AE like cloud-init but
> only purpose is to get files from z/VM internal pipe
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Added a update to the spec to the issues that requested
https://review.openstack.org/#/c/562154/ , including:
1) How the config drive (Metadata) defined
2) Special AE reason and why it's needed, also ,some documentation and
source code links
3) neutron agent for z/VM
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: melanie witt
To: Dan Smith
Cc: openstack-dev@lists.openstack.org
Date: 04/18/2018 01:47 AM
Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
On Tue, 17 Apr 2018 06:40:35 -0700, Dan Smith wrote:
>> I propose that we remove the z/VM driver blueprint from the runway at
>> this time and place it back into the queue while work on the driver
>> continues. At a minimum, we need to see z/VM CI running with
>> [validation]run_validation = True in tempest.conf before we add the
>> z/VM driver blueprint back into a runway in the future.
>
> Agreed. I also want to see the CI reporting cleaned up so that it's
> readable and consistent. Yesterday I pointed out some issues with the
> fact that the actual config files being used are not the ones being
> uploaded. There are also duplicate (but not actually identical) logs
> from all services being uploaded, including things like a full compute
> log from starting with the libvirt driver.
Yes, we definitely need to see all of these issues fixed.
> I'm also pretty troubled by the total lack of support for the metadata
> service. I know it's technically optional on our matrix, but it's a
> pretty important feature for a lot of scenarios, and it's also a
> dependency for other features that we'd like to have wider support for
> (like attached device metadata).
>
> Going back to the spec, I see very little detail on some of the things
> raised here, and very (very) little review back when it was first
> approved. I'd also like to see more detail be added to the spec about
> all of these things, especially around required special changes like
> this extra AE agent.
Agreed, can someone from the z/VM team please propose an update to the
driver spec to document these details?
Thanks,
-melanie
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=CCtWdN4OOlqKzrLg4ctuY1D_fHo8wvps59hVs35J8ys&s=wHuQV89_dwXLe15VAkg8_UOBPfjD72vB0_47W6BgRVk&e=
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
I'm confused about the design of AE to be honest. Is there a good reason that this functionality couldn't be provided by cloud-init? I think there's a lot of cost in deviating from the industry standard, so the reasons to do so have to be really solid. I'm also a bit confused by what seems to be support for streaming configuration. Is there any documentation on the design of AE anywhere? Thanks, Michael On Tue, Apr 17, 2018 at 6:58 PM, Chen CH Ji wrote: > For the question on AE documentation, it's open source in [1] and the > documentation for how to build and use is [2] > once our code is upstream, there are a set of documentation change which > will cover this image build process by > adding some links to there [3] > > You are right, we need image to have our Active Engine, I think different > arch and platform might have their unique > requirements and our solution our Active Engine is very like to cloud-init > so no harm to add it from user's perspective > I think later we can upload image to some place so anyone is able to > consume it as test image if they like > because different arch's image (e.g x86 and s390x) can't be shared anyway. > > For the config drive format you mentioned, actually, as previous > explanation and discussion witho Michael and Dan, > We found the iso9660 can be used (previously we made a bad assumption) and > we already changed the patch in [4], > so it's exactly same to other virt drivers you mentioned , we don't need > special format and iso9660 works perfect for our driver > > It make sense to me we are temply moved out from runway, I suppose we can > adjust the CI to enable the run_ssh = true > with config drive functionalities very soon and we will apply for review > after that with the test result requested in our CI log. > > Thanks > > [1] https://github.com/mfcloud/python-zvm-sdk/blob/master/ > tools/share/zvmguestconfigure > [2] http://cloudlib4zvm.readthedocs.io/en/latest/ > makeimage.html#configuration-of-activation-engine-ae-in-zlinux > [3] https://review.openstack.org/#/q/status:open+project: > openstack/nova+branch:master+topic:bp/add-zvm-driver-rocky > [4] https://review.openstack.org/#/c/527658/33/nova/virt/zvm/utils.py > line 104 > > Best Regards! > > Kevin (Chen) Ji 纪 晨 > > Engineer, zVM Development, CSTL > Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com > Phone: +86-10-82451493 > Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, > Beijing 100193, PRC > > [image: Inactive hide details for melanie witt ---04/17/2018 09:21:03 > AM---On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote: > >>>]melanie > witt ---04/17/2018 09:21:03 AM---On Mon, 16 Apr 2018 14:56:06 +0800, Chen > Ch Ji wrote: > >>>The "iso file" will not be inside the gu > > From: melanie witt > To: openstack-dev@lists.openstack.org > Date: 04/17/2018 09:21 AM > Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config > driveformat > -- > > > > On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote: > > >>>The "iso file" will not be inside the guest, but rather passed to > > the guest as a block device, right? > > Cloud init expects to find a config drive with following requirements > > [1], in order to make cloud init able to consume config drive , we > > should be able to prepare it, > > in some hypervisor, you can define something like following to the VM > > then VM startup is able to consume it > > > > but for z/VM case it allows disk to be created during VM create (define > > )stage but no disk format set, it's the operating system's > > responsibility to define the purpose of the > > disk, so what we do is > > 1) first when we build image ,we create a small AE like cloud-init but > > only purpose is to get files from z/VM internal pipe and handle config > > drive case > > What does AE stand for? So, this means in order to use the z/VM driver, > users must have special images that will ensure the config drive will be > readable by cloud-init. They can't use standard cloud images. > > > 2) During spawn we create config drive in nova-compute side then send > > the file to z/VM through z/VM internal pipe (omit detail here) > > 3) During startup of the virtual machine, the small AE is able to mount > > the file as loop device and then in turn cloud-init is able to handle it > > > > because this is our special case, we don't want to upload to cloud-init > > community because of uniqueness and as far as we can tell, no hook in > &g
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Tue, 17 Apr 2018 06:40:35 -0700, Dan Smith wrote: I propose that we remove the z/VM driver blueprint from the runway at this time and place it back into the queue while work on the driver continues. At a minimum, we need to see z/VM CI running with [validation]run_validation = True in tempest.conf before we add the z/VM driver blueprint back into a runway in the future. Agreed. I also want to see the CI reporting cleaned up so that it's readable and consistent. Yesterday I pointed out some issues with the fact that the actual config files being used are not the ones being uploaded. There are also duplicate (but not actually identical) logs from all services being uploaded, including things like a full compute log from starting with the libvirt driver. Yes, we definitely need to see all of these issues fixed. I'm also pretty troubled by the total lack of support for the metadata service. I know it's technically optional on our matrix, but it's a pretty important feature for a lot of scenarios, and it's also a dependency for other features that we'd like to have wider support for (like attached device metadata). Going back to the spec, I see very little detail on some of the things raised here, and very (very) little review back when it was first approved. I'd also like to see more detail be added to the spec about all of these things, especially around required special changes like this extra AE agent. Agreed, can someone from the z/VM team please propose an update to the driver spec to document these details? Thanks, -melanie __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Tue, 17 Apr 2018 16:58:22 +0800, Chen Ch Ji wrote: For the question on AE documentation, it's open source in [1] and the documentation for how to build and use is [2] once our code is upstream, there are a set of documentation change which will cover this image build process by adding some links to there [3] Thanks, that is good info. You are right, we need image to have our Active Engine, I think different arch and platform might have their unique requirements and our solution our Active Engine is very like to cloud-init so no harm to add it from user's perspective I think later we can upload image to some place so anyone is able to consume it as test image if they like because different arch's image (e.g x86 and s390x) can't be shared anyway. For the config drive format you mentioned, actually, as previous explanation and discussion witho Michael and Dan, We found the iso9660 can be used (previously we made a bad assumption) and we already changed the patch in [4], so it's exactly same to other virt drivers you mentioned , we don't need special format and iso9660 works perfect for our driver That's good news, I'm glad that got resolved. It make sense to me we are temply moved out from runway, I suppose we can adjust the CI to enable the run_ssh = true with config drive functionalities very soon and we will apply for review after that with the test result requested in our CI log. Okay, sounds good. Since you expect to be up and running with [validation]run_validation = True soon, I'm going to move the z/VM driver blueprint back to the front of the queue and put the next blueprint in line into the runway. Then, when the next blueprint end date arrives (currently 2018-04-30), if the z/VM CI is ready with cleaned up, human readable log files and is running with run_ssh = True with the test_server_basic_ops test to verify config drive operation, we will add the z/VM driver blueprint back to a runway for dedicated review. Let us know when the z/VM CI is ready, in case other runway reviews are completed early. If other runway reviews complete early, a runway space might be available earlier than 2018-04-30. Thanks, -melanie Thanks [1] https://github.com/mfcloud/python-zvm-sdk/blob/master/tools/share/zvmguestconfigure [2] http://cloudlib4zvm.readthedocs.io/en/latest/makeimage.html#configuration-of-activation-engine-ae-in-zlinux [3] https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/add-zvm-driver-rocky [4] https://review.openstack.org/#/c/527658/33/nova/virt/zvm/utils.pyline 104 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> I propose that we remove the z/VM driver blueprint from the runway at > this time and place it back into the queue while work on the driver > continues. At a minimum, we need to see z/VM CI running with > [validation]run_validation = True in tempest.conf before we add the > z/VM driver blueprint back into a runway in the future. Agreed. I also want to see the CI reporting cleaned up so that it's readable and consistent. Yesterday I pointed out some issues with the fact that the actual config files being used are not the ones being uploaded. There are also duplicate (but not actually identical) logs from all services being uploaded, including things like a full compute log from starting with the libvirt driver. I'm also pretty troubled by the total lack of support for the metadata service. I know it's technically optional on our matrix, but it's a pretty important feature for a lot of scenarios, and it's also a dependency for other features that we'd like to have wider support for (like attached device metadata). Going back to the spec, I see very little detail on some of the things raised here, and very (very) little review back when it was first approved. I'd also like to see more detail be added to the spec about all of these things, especially around required special changes like this extra AE agent. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On 04/16/2018 09:20 PM, melanie witt wrote: I propose that we remove the z/VM driver blueprint from the runway at this time and place it back into the queue while work on the driver continues. At a minimum, we need to see z/VM CI running with [validation]run_validation = True in tempest.conf before we add the z/VM driver blueprint back into a runway in the future. Seems reasonable to me. -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
For the question on AE documentation, it's open source in [1] and the documentation for how to build and use is [2] once our code is upstream, there are a set of documentation change which will cover this image build process by adding some links to there [3] You are right, we need image to have our Active Engine, I think different arch and platform might have their unique requirements and our solution our Active Engine is very like to cloud-init so no harm to add it from user's perspective I think later we can upload image to some place so anyone is able to consume it as test image if they like because different arch's image (e.g x86 and s390x) can't be shared anyway. For the config drive format you mentioned, actually, as previous explanation and discussion witho Michael and Dan, We found the iso9660 can be used (previously we made a bad assumption) and we already changed the patch in [4], so it's exactly same to other virt drivers you mentioned , we don't need special format and iso9660 works perfect for our driver It make sense to me we are temply moved out from runway, I suppose we can adjust the CI to enable the run_ssh = true with config drive functionalities very soon and we will apply for review after that with the test result requested in our CI log. Thanks [1] https://github.com/mfcloud/python-zvm-sdk/blob/master/tools/share/zvmguestconfigure [2] http://cloudlib4zvm.readthedocs.io/en/latest/makeimage.html#configuration-of-activation-engine-ae-in-zlinux [3] https://review.openstack.org/#/q/status:open+project:openstack/nova +branch:master+topic:bp/add-zvm-driver-rocky [4] https://review.openstack.org/#/c/527658/33/nova/virt/zvm/utils.py line 104 Best Regards! Kevin (Chen) Ji 纪 晨 Engineer, zVM Development, CSTL Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com Phone: +86-10-82451493 Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC From: melanie witt To: openstack-dev@lists.openstack.org Date: 04/17/2018 09:21 AM Subject: Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote: > >>>The "iso file" will not be inside the guest, but rather passed to > the guest as a block device, right? > Cloud init expects to find a config drive with following requirements > [1], in order to make cloud init able to consume config drive , we > should be able to prepare it, > in some hypervisor, you can define something like following to the VM > then VM startup is able to consume it > > but for z/VM case it allows disk to be created during VM create (define > )stage but no disk format set, it's the operating system's > responsibility to define the purpose of the > disk, so what we do is > 1) first when we build image ,we create a small AE like cloud-init but > only purpose is to get files from z/VM internal pipe and handle config > drive case What does AE stand for? So, this means in order to use the z/VM driver, users must have special images that will ensure the config drive will be readable by cloud-init. They can't use standard cloud images. > 2) During spawn we create config drive in nova-compute side then send > the file to z/VM through z/VM internal pipe (omit detail here) > 3) During startup of the virtual machine, the small AE is able to mount > the file as loop device and then in turn cloud-init is able to handle it > > because this is our special case, we don't want to upload to cloud-init > community because of uniqueness and as far as we can tell, no hook in > cloud-init mechanism allowed as well > to let us 'mount -o loop' ; also, from openstack point of view except > this small AE (which is documented well) no special thing and > inconsistent to other drivers > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_number5_cloud-2Dinit_blob_master_cloudinit_sources_DataSourceConfigDrive.py-23L225&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=yV6OJ4IfFSLoHNWAJpBF7j0sK2pgfxSEIigv8vinYw0&s=3410axnNZ_62U3HOh6i7yivyc7HyTcqwx2xuKRDEeac&e= Where is the AE documented? How do users obtain it? What tools are they supposed to use to build images to use with the z/VM driver? That aside, from what I can see, the z/VM driver behaves unlike any other in-tree driver [0-5] in how it handles config drive. Drivers are expected to create the config drive and present it to the guest in iso9660 or vfat format without requirement of a custom image and the existing drivers are doing that. IMHO, if the z/VM driver can't be fixed to provide proper config drive support, we won't be able to approve the implementation patches. I would like to hear other opinions about it. I propose that we remove the z/VM driv
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
On Mon, 16 Apr 2018 14:56:06 +0800, Chen Ch Ji wrote: >>>The "iso file" will not be inside the guest, but rather passed to the guest as a block device, right? Cloud init expects to find a config drive with following requirements [1], in order to make cloud init able to consume config drive , we should be able to prepare it, in some hypervisor, you can define something like following to the VM then VM startup is able to consume it but for z/VM case it allows disk to be created during VM create (define )stage but no disk format set, it's the operating system's responsibility to define the purpose of the disk, so what we do is 1) first when we build image ,we create a small AE like cloud-init but only purpose is to get files from z/VM internal pipe and handle config drive case What does AE stand for? So, this means in order to use the z/VM driver, users must have special images that will ensure the config drive will be readable by cloud-init. They can't use standard cloud images. 2) During spawn we create config drive in nova-compute side then send the file to z/VM through z/VM internal pipe (omit detail here) 3) During startup of the virtual machine, the small AE is able to mount the file as loop device and then in turn cloud-init is able to handle it because this is our special case, we don't want to upload to cloud-init community because of uniqueness and as far as we can tell, no hook in cloud-init mechanism allowed as well to let us 'mount -o loop' ; also, from openstack point of view except this small AE (which is documented well) no special thing and inconsistent to other drivers [1]https://github.com/number5/cloud-init/blob/master/cloudinit/sources/DataSourceConfigDrive.py#L225 Where is the AE documented? How do users obtain it? What tools are they supposed to use to build images to use with the z/VM driver? That aside, from what I can see, the z/VM driver behaves unlike any other in-tree driver [0-5] in how it handles config drive. Drivers are expected to create the config drive and present it to the guest in iso9660 or vfat format without requirement of a custom image and the existing drivers are doing that. IMHO, if the z/VM driver can't be fixed to provide proper config drive support, we won't be able to approve the implementation patches. I would like to hear other opinions about it. I propose that we remove the z/VM driver blueprint from the runway at this time and place it back into the queue while work on the driver continues. At a minimum, we need to see z/VM CI running with [validation]run_validation = True in tempest.conf before we add the z/VM driver blueprint back into a runway in the future. Cheers, -melanie [0] https://github.com/openstack/nova/blob/888cd51/nova/virt/hyperv/vmops.py#L661 [1] https://github.com/openstack/nova/blob/888cd51/nova/virt/ironic/driver.py#L974 [2] https://github.com/openstack/nova/blob/888cd51/nova/virt/libvirt/driver.py#L3595 [3] https://github.com/openstack/nova/blob/888cd51/nova/virt/powervm/media.py#L120 [4] https://github.com/openstack/nova/blob/888cd51/nova/virt/vmwareapi/vmops.py#L854 [5] https://github.com/openstack/nova/blob/888cd51/nova/virt/xenapi/vm_utils.py#L1151 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Thanks for the question and comments
>>> metadata service question
Fully agree metadata is something that we need support, and as it need some
network setup on NAT ,as you pointed out, without metadata there are some
functions missing ; so it's already in our support plan , currently we plan
to use config drive and later (with the enhance with our neutron support as
well) to support metadata service
>>>The "iso file" will not be inside the guest, but rather passed to the
guest as a block device, right?
Cloud init expects to find a config drive with following requirements [1],
in order to make cloud init able to consume config drive , we should be
able to prepare it,
in some hypervisor, you can define something like following to the VM then
VM startup is able to consume it
but for z/VM case it allows disk to be created during VM create
(define )stage but no disk format set, it's the operating system's
responsibility to define the purpose of the
disk, so what we do is
1) first when we build image ,we create a small AE like cloud-init but only
purpose is to get files from z/VM internal pipe and handle config drive
case
2) During spawn we create config drive in nova-compute side then send the
file to z/VM through z/VM internal pipe (omit detail here)
3) During startup of the virtual machine, the small AE is able to mount the
file as loop device and then in turn cloud-init is able to handle it
because this is our special case, we don't want to upload to cloud-init
community because of uniqueness and as far as we can tell, no hook in
cloud-init mechanism allowed as well
to let us 'mount -o loop' ; also, from openstack point of view except this
small AE (which is documented well) no special thing and inconsistent to
other drivers
[1]
https://github.com/number5/cloud-init/blob/master/cloudinit/sources/DataSourceConfigDrive.py#L225
Best Regards!
Kevin (Chen) Ji 纪 晨
Engineer, zVM Development, CSTL
Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com
Phone: +86-10-82451493
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District,
Beijing 100193, PRC
From: Dan Smith
To: "Chen CH Ji"
Cc: "OpenStack Development Mailing List \(not for usage questions
\)"
Date: 04/13/2018 09:46 PM
Subject:Re: [openstack-dev] [Nova] z/VM introducing a new config
driveformat
> for the run_validation=False issue, you are right, because z/VM driver
> only support config drive and don't support metadata service ,we made
> bad assumption and took wrong action to disabled the whole ssh check,
> actually according to [1] , we should only disable
> CONF.compute_feature_enabled.metadata_service but keep both
> self.run_ssh and CONF.compute_feature_enabled.config_drive as True in
> order to make config drive test validation take effect, our CI will
> handle that
Why don't you support the metadata service? That's a pretty fundamental
mechanism for nova and openstack. It's the only way you can get a live
copy of metadata, and it's the only way you can get access to device
tags when you hot-attach something. Personally, I think that it's
something that needs to work.
> For the tgz/iso9660 question below, this is because we got wrong info
> from low layer component folks back to 2012 and after discuss with
> some experts again, actually we can create iso9660 in the driver layer
> and pass down to the spawned virtual machine and during startup
> process, the VM itself will mount the iso file and consume it, because
> from linux perspective, either tgz or iso9660 doesn't matter , only
> need some files in order to transfer the information from openstack
> compute node to the spawned VM. so our action is to change the format
> from tgz to iso9660 and keep consistent to other drivers.
The "iso file" will not be inside the guest, but rather passed to the
guest as a block device, right?
> For the config drive working mechanism question, according to [2] z/VM
> is Type 1 hypervisor while Qemu/KVM are mostly likely to be Type 2
> hypervisor, there is no file system in z/VM hypervisor (I omit too
> much detail here) , so we can't do something like linux operation
> system to keep a file as qcow2 image in the host operating system,
I'm not sure what the type-1-ness has to do with this. The hypervisor
doesn't need to support any specific filesystem for this to work. Many
drivers we have in the tree are type-1 (xen, vmware, hyperv, powervm)
and you can argue that KVM is type-1-ish. They support configdrive.
> what we do is use a special file pool to store the config drive and
> during VM init process, we read that file from special device and
> attach to VM as iso9660 format then cloud-init will handle the follow
> up, the cloud-init handle process is
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
> for the run_validation=False issue, you are right, because z/VM driver > only support config drive and don't support metadata service ,we made > bad assumption and took wrong action to disabled the whole ssh check, > actually according to [1] , we should only disable > CONF.compute_feature_enabled.metadata_service but keep both > self.run_ssh and CONF.compute_feature_enabled.config_drive as True in > order to make config drive test validation take effect, our CI will > handle that Why don't you support the metadata service? That's a pretty fundamental mechanism for nova and openstack. It's the only way you can get a live copy of metadata, and it's the only way you can get access to device tags when you hot-attach something. Personally, I think that it's something that needs to work. > For the tgz/iso9660 question below, this is because we got wrong info > from low layer component folks back to 2012 and after discuss with > some experts again, actually we can create iso9660 in the driver layer > and pass down to the spawned virtual machine and during startup > process, the VM itself will mount the iso file and consume it, because > from linux perspective, either tgz or iso9660 doesn't matter , only > need some files in order to transfer the information from openstack > compute node to the spawned VM. so our action is to change the format > from tgz to iso9660 and keep consistent to other drivers. The "iso file" will not be inside the guest, but rather passed to the guest as a block device, right? > For the config drive working mechanism question, according to [2] z/VM > is Type 1 hypervisor while Qemu/KVM are mostly likely to be Type 2 > hypervisor, there is no file system in z/VM hypervisor (I omit too > much detail here) , so we can't do something like linux operation > system to keep a file as qcow2 image in the host operating system, I'm not sure what the type-1-ness has to do with this. The hypervisor doesn't need to support any specific filesystem for this to work. Many drivers we have in the tree are type-1 (xen, vmware, hyperv, powervm) and you can argue that KVM is type-1-ish. They support configdrive. > what we do is use a special file pool to store the config drive and > during VM init process, we read that file from special device and > attach to VM as iso9660 format then cloud-init will handle the follow > up, the cloud-init handle process is identical to other platform This and the previous mention of this sort of behavior has me concerned. Are you describing some sort of process that runs when the instance is starting to initialize its environment, or something that runs *inside* the instance and thus functionality that has to exist in the *image* to work? --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] z/VM introducing a new config driveformat
Thanks for raising this question, really helpful to the enhancement of our driver for the run_validation=False issue, you are right, because z/VM driver only support config drive and don't support metadata service ,we made bad assumption and took wrong action to disabled the whole ssh check, actually according to [1] , we should only disable CONF.compute_feature_enabled.metadata_service but keep both self.run_ssh and CONF.compute_feature_enabled.config_drive as True in order to make config drive test validation take effect, our CI will handle that For the tgz/iso9660 question below, this is because we got wrong info from low layer component folks back to 2012 and after discuss with some experts again, actually we can create iso9660 in the driver layer and pass down to the spawned virtual machine and during startup process, the VM itself will mount the iso file and consume it, because from linux perspective, either tgz or iso9660 doesn't matter , only need some files in order to transfer the information from openstack compute node to the spawned VM. so our action is to change the format from tgz to iso9660 and keep consistent to other drivers. For the config drive working mechanism question, according to [2] z/VM is Type 1 hypervisor while Qemu/KVM are mostly likely to be Type 2 hypervisor, there is no file system in z/VM hypervisor (I omit too much detail here) , so we can't do something like linux operation system to keep a file as qcow2 image in the host operating system, what we do is use a special file pool to store the config drive and during VM init process, we read that file from special device and attach to VM as iso9660 format then cloud-init will handle the follow up, the cloud-init handle process is identical to other platform Again, The tgz/iso9660 format is only because tgz format being wrongly thought, we already have some existing customers and a public openstack cloud [3] running on LinuxONE (system z) [4] so config drive of z/VM driver does work and we will modify our code to be consistent to community in our patch set please let us know any further question, thanks [1] https://github.com/openstack/tempest/blob/master/tempest/scenario/test_server_basic_ops.py#L68 [2]https://en.wikipedia.org/wiki/Hypervisor [3]https://linuxone20.cloud.marist.edu/cloud/ [4]https://www.zdnet.com/article/linuxone-ibms-new-linux-mainframes/ Best Regards! Kevin (Chen) Ji 纪 晨 Engineer, zVM Development, CSTL Notes: Chen CH Ji/China/IBM@IBMCN Internet: jiche...@cn.ibm.com Phone: +86-10-82451493 Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC From: melanie witt To: openstack-dev@lists.openstack.org Date: 04/13/2018 03:39 AM Subject:Re: [openstack-dev] [Nova] z/VM introducing a new config drive format On Thu, 12 Apr 2018 09:31:45 +1000, Michael Still wrote: > The more I think about it, the more I dislike how the proposed driver > also "lies" about it using iso9660. That's definitely wrong: > > if CONF.config_drive_format in ['iso9660']: > # cloud-init only support iso9660 and vfat, but in z/VM > # implementation, can't link a disk to VM as iso9660 before > it's > # boot ,so create a tgz file then send to the VM deployed, and > # during startup process, the tgz file will be extracted and > # mounted as iso9660 format then cloud-init is able to > consume it > self._make_tgz(path) > else: > raise exception.ConfigDriveUnknownFormat( > format=CONF.config_drive_format) I've asked for more information on the review about how this works -- is it the z/VM library that extracts the tarball and mounts it as iso9660 before the guest boots or is it expected that the guest is running some special software that will do that before cloud-init runs, or what? I also noticed that the z/VM CI has disabled ssh validation of guests by setting '[validation]run_validation=False' in tempest.conf [0]. This means we're unable to see the running guest successfully consume the config drive using this approach. This is the tempest test that verifies functionality when run_validation=True [1]. I think we need to understand more about how this config drive approach is supposed to work and be able to see running instances successfully start up using it in the CI runs. -melanie [0] http://extbasicopstackcilog01.podc.sl.edst.ibm.com/test_logs/jenkins-check-nova-master-16244/logs/tempest_conf [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_tempest_blob_master_tempest_scenario_test-5Fserver-5Fbasic-5Fops.py&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=8sI5aZT88Uetyy_XsOddbPjIiLSGM-sFnua3lLy2Xr0&m=d9VEfLe0LqlPnfL0F0DwKa5iNpsRfDQKiobInGR02lc&s=0X5hrQ3zh7vwq7wJJAbox4M_4p0myAC1zehbbxYGNF8&e= __ OpenStack Development Mailing List (not for usa
