On 12/09/2014 03:40 PM, Vladimir Kozhukalov wrote:
Vladimir Kozhukalov
On Tue, Dec 9, 2014 at 3:51 PM, Dmitry Tantsur <dtant...@redhat.com
<mailto:dtant...@redhat.com>> wrote:
Hi folks,
Thank you for additional explanation, it does clarify things a bit.
I'd like to note, however, that you talk a lot about how _different_
Fuel Agent is from what Ironic does now. I'd like actually to know
how well it's going to fit into what Ironic does (in additional to
your specific use cases). Hence my comments inline:
On 12/09/2014 01:01 PM, Vladimir Kozhukalov wrote:
Just a short explanation of Fuel use case.
Fuel use case is not a cloud. Fuel is a deployment tool. We
install OS
on bare metal servers and on VMs
and then configure this OS using Puppet. We have been using
Cobbler as
our OS provisioning tool since the beginning of Fuel.
However, Cobbler assumes using native OS installers (Anaconda and
Debian-installer). For some reasons we decided to
switch to image based approach for installing OS.
One of Fuel features is the ability to provide advanced partitioning
schemes (including software RAIDs, LVM).
Native installers are quite difficult to customize in the field of
partitioning
(that was one of the reasons to switch to image based approach).
Moreover, we'd like to implement even more
flexible user experience. We'd like to allow user to choose
which hard
drives to use for root FS, for
allocating DB. We'd like user to be able to put root FS over LV
or MD
device (including stripe, mirror, multipath).
We'd like user to be able to choose which hard drives are
bootable (if
any), which options to use for mounting file systems.
Many many various cases are possible. If you ask why we'd like to
support all those cases, the answer is simple:
because our users want us to support all those cases.
Obviously, many of those cases can not be implemented as image
internals, some cases can not be also implemented on
configuration stage (placing root fs on lvm device).
As far as those use cases were rejected to be implemented in term of
IPA, we implemented so called Fuel Agent.
Important Fuel Agent features are:
* It does not have REST API
I would not call it a feature :-P
Speaking seriously, if you agent is a long-running thing and it gets
it's configuration from e.g. JSON file, how can Ironic notify it of
any changes?
Fuel Agent is not long-running service. Currently there is no need to
have REST API. If we deal with kind of keep alive stuff of
inventory/discovery then we probably add API. Frankly, IPA REST API is
not REST at all. However that is not a reason to not to call it a
feature and through it away. It is a reason to work on it and improve.
That is how I try to look at things (pragmatically).
Fuel Agent has executable entry point[s] like /usr/bin/provision. You
can run this entry point with options (oslo.config) and point out where
to find input json data. It is supposed Ironic will use ssh (currently
in Fuel we use mcollective) connection and run this waiting for exit
code. If exit code is equal to 0, provisioning is done. Extremely simple.
* it has executable entry point[s]
* It uses local json file as it's input
* It is planned to implement ability to download input data via HTTP
(kind of metadata service)
* It is designed to be agnostic to input data format, not only Fuel
format (data drivers)
* It is designed to be agnostic to image format (tar images,
file system
images, disk images, currently fs images)
* It is designed to be agnostic to image compression algorithm
(currently gzip)
* It is designed to be agnostic to image downloading protocol
(currently
local file and HTTP link)
Does it support Glance? I understand it's HTTP, but it requires
authentication.
So, it is clear that being motivated by Fuel, Fuel Agent is quite
independent and generic. And we are open for
new use cases.
My favorite use case is hardware introspection (aka getting data
required for scheduling from a node automatically). Any ideas on
this? (It's not a priority for this discussion, just curious).
That is exactly what we do in Fuel. Currently we use so called 'Default'
pxelinux config and all nodes being powered on are supposed to boot with
so called 'Bootstrap' ramdisk where Ohai based agent (not Fuel Agent)
runs periodically and sends hardware report to Fuel master node.
User then is able to look at CPU, hard drive and network info and choose
which nodes to use for controllers, which for computes, etc. That is
what nova scheduler is supposed to do (look at hardware info and choose
a suitable node).
Talking about future, we are planning to re-implement
inventory/discovery stuff in terms of Fuel Agent (currently, this stuff
is implemented as Ohai based independent script). Estimation for that
is March 2015.
According Fuel itself, our nearest plan is to get rid of Cobbler
because
in the case of image based approach it is huge overhead. The
question is
which tool we can use instead of Cobbler. We need power management,
we need TFTP management, we need DHCP management. That is
exactly what Ironic is able to do. Frankly, we can implement
power/TFTP/DHCP
management tool independently, but as Devananda said, we're all
working
on the same problems,
so let's do it together. Power/TFTP/DHCP management is where we are
working on the same problems,
but IPA and Fuel Agent are about different use cases. This case
is not
just Fuel, any mature
deployment case require advanced partition/fs management.
Taking into consideration that you're doing a generic OS
installation tool... yeah, it starts to make some sense. For cloud
advanced partition is definitely a "pet" case.
Generic image based OS installation tool.
However, for
me it is OK, if it is easily possible
to use Ironic with external drivers (not merged to Ironic and
not tested
on Ironic CI).
AFAIU, this spec https://review.openstack.org/#__/c/138115/
<https://review.openstack.org/#/c/138115/> does not
assume changing Ironic API and core.
Jim asked about how Fuel Agent will know about advanced disk
partitioning scheme if API is not
changed. The answer is simple: Ironic is supposed to send a link to
metadata service (http or local file)
where Fuel Agent can download input json data.
That's not about not changing Ironic. Changing Ironic is ok for
reasonable use cases - we do a huge change right now to accommodate
zapping, hardware introspection and RAID configuration.
Minimal changes because we don't want to break anything. It is clear how
difficult to convince people to do even minimal changes. Again it is
just a pragmatic approach. We want to do things iteratively so as not
to break Ironic as well as Fuel. We just can not change all at once.
I actually have problems with this particular statement. It does not
sound like Fuel Agent will integrate enough with Ironic. This JSON
file: who is going to generate it? In the most popular use case
we're driven by Nova. Will Nova generate this file?
If the answer is "generate it manually for every node", it's too
much a "pet" case for me personally.
That is how this provision data look like right now
https://github.com/stackforge/fuel-web/blob/master/fuel_agent_ci/samples/provision.json
Do you still think it is written manually? Currently Fuel Agent works
as a part of Fuel ecosystem. We have a service which serializes
provision data for us into json. Fuel Agent is agnostic to data format
(data drivers). If someone wants to use another format, they are welcome
to implement a driver.
We assume next step will be to put provision data (disk partition
scheme, maybe other data) into driver_info and make Fuel Agent driver
able to serialize those data (special format) and implement a
corresponding data driver in Fuel Agent for this format. Again very
simple. Maybe it is time to think of having Ironic metadata service
(just maybe).
I'm ok with the format, my question is: what and how is going to collect
all the data and put into say driver_info?
Another point is that currently Fuel stores hardware info in its own
database but when it is possible to get those data from Ironic (when
inventory stuff is implemented) we will be glad to use Ironic API for
that. That is what I mean when I say 'to make Fuel stuff closer to
Ironic abstractions'
As Roman said, we try to be pragmatic and suggest something
which does
not break anything. All changes
are supposed to be encapsulated into a driver. No API and core
changes.
We have resources to support, test
and improve this driver. This spec is just a zero step. Further
steps
are supposed to improve driver
so as to make it closer to Ironic abstractions.
Honestly I think you should at least write a roadmap for it - see my
comments above.
Honestly, I think writing roadmap right now is not very rational as far
as I am not even sure people are interested in widening Ironic use
cases. Some of the comments were not even constructive like "I don't
understand what your use case is, please use IPA".
Please don't be offended by this. We did put a lot of effort into IPA
and it's reasonable to look for a good use cases before having one more
smart ramdisk. Nothing personal, just estimating cost vs value :)
Also "why not use IPA" is a fair question for me and the answer is about
use cases (as you stated it before), not about missing features of IPA,
right?
About testing and support: are you providing a 3rdparty CI for it?
It would be a big plus as to me: we already have troubles with
drivers broken accidentally.
We are flexible here but I'm not ready to answer this question right
now. We will try to fit Ironic requirements wherever it is possible.
For Ironic that means widening use cases and user community.
But, as I
already said,
we are OK if Ironic does not need this feature.
I don't think we should through away your hardware provision use
case, but I personally would like to see how well Fuel Agent is
going to play with how Ironic and Nova operate.
Nova is not our case. Fuel is totally about deployment. There is some in
common
Here when we have a difficult point. Major use case for Ironic is to be
driven by Nova (and assisted by Neutron). Without these two it's hard to
understand how Fuel Agent is going to fit into the infrastructure. And
hence my question above about where your json comes from. In the current
Ironic world the same data is received partly from Nova flavor, partly
managed by Neutron completely.
I'm not saying it can't change - we do want to become more stand-alone.
E.g. we can do without Neutron right now. I think specifying the source
of input data for Fuel Agent in the Ironic infrastructure would help a
lot understand, how well Ironic and Fuel Agent could play together.
As I already explained, currently we need power/tftp/dhcp management
Ironic capabilities. Again, it is not a problem to implement this stuff
independently like it happened with Fuel Agent (because this use case
was rejected several months ago). Our suggestion is not about "let's
compete with IPA" it is totally about "let's work on the same problems
together".
Vladimir Kozhukalov
On Tue, Dec 9, 2014 at 1:09 PM, Roman Prykhodchenko
<rprikhodche...@mirantis.com
<mailto:rprikhodche...@mirantis.com>
<mailto:rprikhodchenko@__mirantis.com
<mailto:rprikhodche...@mirantis.com>>> wrote:
It is true that IPA and FuelAgent share a lot of
functionality in
common. However there is a major difference between them
which is
that they are intended to be used to solve a different problem.
IPA is a solution for
provision-use-destroy-use_by___different_user
use-case and is really great for using it for providing BM
nodes for
other OS services or in services like Rackspace OnMetal.
FuelAgent
itself serves for provision-use-use-…-use use-case like Fuel or
TripleO have.
Those two use-cases require concentration on different
details in
first place. For instance for IPA proper decommissioning is
more
important than advanced disk management, but for FuelAgent
priorities are opposite because of obvious reasons.
Putting all functionality to a single driver and a single
agent may
cause conflicts in priorities and make a lot of mess inside
both the
driver and the agent. Actually previously changes to IPA were
blocked right because of this conflict of priorities. Therefore
replacing FuelAgent by IPA in where FuelAgent is used
currently does
not seem like a good option because come people (and I’m
not talking
about Mirantis) might loose required features because of
different
priorities.
Having two separate drivers along with two separate agents
for those
different use-cases will allow to have two independent
teams that
are concentrated on what’s really important for a specific
use-case.
I don’t see any problem in overlapping functionality if
it’s used
differently.
P. S.
I realise that people may be also confused by the fact that
FuelAgent is actually called like that and is used only in
Fuel atm.
Our point is to make it a simple, powerful and what’s more
important
a generic tool for provisioning. It is not bound to Fuel or
Mirantis
and if it will cause confusion in the future we will even
be happy
to give it a different and less confusing name.
P. P. S.
Some of the points of this integration do not look generic
enough or
nice enough. We look pragmatic on the stuff and are trying to
implement what’s possible to implement as the first step.
For sure
this is going to have a lot more steps to make it better
and more
generic.
On 09 Dec 2014, at 01:46, Jim Rollenhagen
<j...@jimrollenhagen.com <mailto:j...@jimrollenhagen.com>
<mailto:j...@jimrollenhagen.com
<mailto:j...@jimrollenhagen.com>__>> wrote:
On December 8, 2014 2:23:58 PM PST, Devananda van der Veen
<devananda....@gmail.com
<mailto:devananda....@gmail.com>
<mailto:devananda.vdv@gmail.__com
<mailto:devananda....@gmail.com>>> wrote:
I'd like to raise this topic for a wider discussion
outside of the
hallway
track and code reviews, where it has thus far
mostly remained.
In previous discussions, my understanding has been
that the Fuel team
sought to use Ironic to manage "pets" rather than
"cattle" - and
doing
so
required extending the API and the project's
functionality in
ways that
no
one else on the core team agreed with. Perhaps that
understanding was
wrong
(or perhaps not), but in any case, there is now a
proposal to add a
FuelAgent driver to Ironic. The proposal claims
this would meet that
teams'
needs without requiring changes to the core of Ironic.
https://review.openstack.org/#__/c/138115/
<https://review.openstack.org/#/c/138115/>
I think it's clear from the review that I share the
opinions
expressed in this email.
That said (and hopefully without derailing the thread
too much),
I'm curious how this driver could do software RAID or
LVM without
modifying Ironic's API or data model. How would the
agent know how
these should be built? How would an operator or user
tell Ironic
what the disk/partition/volume layout would look like?
And before it's said - no, I don't think vendor
passthru API calls
are an appropriate answer here.
// jim
The Problem Description section calls out four
things, which have all
been
discussed previously (some are here [0]). I would
like to address
each
one,
invite discussion on whether or not these are, in
fact, problems
facing
Ironic (not whether they are problems for someone,
somewhere),
and then
ask
why these necessitate a new driver be added to the
project.
They are, for reference:
1. limited partition support
2. no software RAID support
3. no LVM support
4. no support for hardware that lacks a BMC
#1.
When deploying a partition image (eg, QCOW format),
Ironic's PXE
deploy
driver performs only the minimal partitioning
necessary to
fulfill its
mission as an OpenStack service: respect the user's
request for root,
swap,
and ephemeral partition sizes. When deploying a
whole-disk image,
Ironic
does not perform any partitioning -- such is left
up to the operator
who
created the disk image.
Support for arbitrarily complex partition layouts
is not required by,
nor
does it facilitate, the goal of provisioning
physical servers via a
common
cloud API. Additionally, as with #3 below, nothing
prevents a
user from
creating more partitions in unallocated disk space
once they have
access to
their instance. Therefor, I don't see how Ironic's
minimal
support for
partitioning is a problem for the project.
#2.
There is no support for defining a RAID in Ironic
today, at all,
whether
software or hardware. Several proposals were
floated last cycle;
one is
under review right now for DRAC support [1], and
there are multiple
call
outs for RAID building in the state machine
mega-spec [2]. Any such
support
for hardware RAID will necessarily be abstract
enough to support
multiple
hardware vendor's driver implementations and both
in-band
creation (via
IPA) and out-of-band creation (via vendor tools).
Given the above, it may become possible to add
software RAID
support to
IPA
in the future, under the same abstraction. This
would closely tie the
deploy agent to the images it deploys (the latter
image's kernel
would
be
dependent upon a software RAID built by the
former), but this would
necessarily be true for the proposed FuelAgent as well.
I don't see this as a compelling reason to add a
new driver to the
project.
Instead, we should (plan to) add support for
software RAID to the
deploy
agent which is already part of the project.
#3.
LVM volumes can easily be added by a user (after
provisioning) within
unallocated disk space for non-root partitions. I
have not yet seen a
compelling argument for doing this within the
provisioning phase.
#4.
There are already in-tree drivers [3] [4] [5] which
do not require a
BMC.
One of these uses SSH to connect and run
pre-determined commands.
Like
the
spec proposal, which states at line 122, "Control
via SSH access
feature
intended only for experiments in non-production
environment," the
current
SSHPowerDriver is only meant for testing
environments. We could
probably
extend this driver to do what the FuelAgent spec
proposes, as far as
remote
power control for cheap always-on hardware in testing
environments with
a
pre-shared key.
(And if anyone wonders about a use case for Ironic
without external
power
control ... I can only think of one situation where
I would
rationally
ever
want to have a control-plane agent running inside a
user-instance: I am
both the operator and the only user of the cloud.)
----------------
In summary, as far as I can tell, all of the
problem statements upon
which
the FuelAgent proposal are based are solvable
through incremental
changes
in existing drivers, or out of scope for the
project entirely. As
another
software-based deploy agent, FuelAgent would
duplicate the
majority of
the
functionality which ironic-python-agent has today.
Ironic's driver ecosystem benefits from a diversity of
hardware-enablement
drivers. Today, we have two divergent software
deployment drivers
which
approach image deployment differently: "agent"
drivers use a local
agent to
prepare a system and download the image; "pxe"
drivers use a remote
agent
and copy the image over iSCSI. I don't understand
how a second driver
which
duplicates the functionality we already have, and
shares the same
goals
as
the drivers we already have, is beneficial to the
project.
Doing the same thing twice just increases the
burden on the team;
we're
all
working on the same problems, so let's do it together.
-Devananda
[0]
https://blueprints.launchpad.__net/ironic/+spec/ironic-__python-agent-partition
<https://blueprints.launchpad.net/ironic/+spec/ironic-python-agent-partition>
[1] https://review.openstack.org/#__/c/107981/
<https://review.openstack.org/#/c/107981/>
[2]
https://review.openstack.org/#__/c/133828/11/specs/kilo/new-__ironic-state-machine.rst
<https://review.openstack.org/#/c/133828/11/specs/kilo/new-ironic-state-machine.rst>
[3]
http://git.openstack.org/cgit/__openstack/ironic/tree/ironic/__drivers/modules/snmp.py
<http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/snmp.py>
[4]
http://git.openstack.org/cgit/__openstack/ironic/tree/ironic/__drivers/modules/iboot.py
<http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/iboot.py>
[5]
http://git.openstack.org/cgit/__openstack/ironic/tree/ironic/__drivers/modules/ssh.py
<http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/ssh.py>
------------------------------__------------------------------__------------
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
<mailto:OpenStack-dev@lists.__openstack.org
<mailto:OpenStack-dev@lists.openstack.org>>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
<mailto:OpenStack-dev@lists.__openstack.org
<mailto:OpenStack-dev@lists.openstack.org>>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
<mailto:OpenStack-dev@lists.__openstack.org
<mailto:OpenStack-dev@lists.openstack.org>>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
