Re: [openstack-dev] [fuel][ConfigDB] Separating node and cluster serialized data

[Oleg Gelbukh]

Bogdan,

I mostly agree with you on this. The only data that might originate from a
node is discovery-related parameters, like CPU/disks/NICs architecture and
such.

However, at the moment the deployment data is partially generated at every
node (i.e. globals.yaml, override/plugins/* and some other files) and is
not exposed in any way externally. But since this data is required to
integrate with 3rd-party configuration management tools, we create an
interim solution to make them available 'as is'.

This situation should change in the next few months, and then nodes shall
be moved to purely consumer role in the deployment data pipeline.

--
Best regards,
Oleg Gelbukh

On Fri, Apr 1, 2016 at 1:37 PM, Bogdan Dobrelya 
wrote:

> On 04/01/2016 10:41 AM, Oleg Gelbukh wrote:
> > Andrew,
> >
> > This is an excellent idea. It is apparently more efficient and
> > error-proof to make the split not by the resulted data but at the time
> > it is actually generated. We will play with this idea a little bit, and
> > will come up with design proposal shortly.
> >
> > Meanwhile, please be informed that we already started testing the
> > solution based on the node-level data exposed via ConfigDB API extension
> > for Nailgun [1] [2]. I will keep you updated on our progress in that
> area.
>
> I strongly believe that nodes must only consume data, not provide one.
> And the data must be collected from its sources, which is Nailgun API
> extensions, like Andrew described.
>
> >
> > [1] Specification for Nailgun API for serialized facts
> > 
> > [2] Spec for upload of deployment configuration to ConfigDB API
> > 
> >
> > --
> > Best regards,
> > Oleg Gelbukh
> >
> > On Thu, Mar 31, 2016 at 11:19 PM, Andrew Woodward  > > wrote:
> >
> > One of the problems we've faced with trying to plug-in ConfigDB is
> > trying to separate the cluster attributes from the node attributes
> > in the serialized output (ie astute.yaml)
> >
> > I started talking with Alex S about how we could separate them after
> > astute.yaml is prepared trying to ensure which was which we came
> > back uncertain that the results would be accurate.
> >
> > So I figured I'd go back to the source and see if there was a way to
> > know which keys belonged where. It turns out that we could solve the
> > problem in a simpler and more precise way than cutting them back
> > apart later.
> >
> > Looking over the deployment_serializers.py [1] the serialized data
> > follows a simple work flow
> >
> > iterate over every node in cluster
> >   if node is customized:
> > serialized_data = node.replaced_deployment_data
> >   else:
> > serialized_data = dict_merge(
> >   serialize_node(node),
> >   get_common_attrs(cluster))
> >
> > Taking this into mind, we can simply construct an extension to
> > expose these as an APIs so that we can consume them as a task in the
> > deployment graph.
> >
> > Cluster:
> > We can simply expose
> > DeploymentMultinodeSerializer().get_common_attrs(cluster)
> >
> > This would then be plumbed to the cluster level in ConfigDB
> >
> > Node:
> > if a Node has customized data, then we can return that at the node
> > level, this continues to work at the same as native since it most
> > likely has Cluster merged into it.
> >
> > otherwise we can return the serialized node with whichever of the
> > first 'role' the node has
> >
> > We would expose DeploymentMultinodeSerializer().serialize_node(node,
> > objects.Node.all_roles(node)[0])
> >
> > for our usage, we don't need to worry about the normal node role
> > combination as the data only influences 'role' and 'fail_if_error'
> > attributes, both are not consumed in the library.
> >
> >
> https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py#L93-L121
> > --
> >
> > --
> >
> > Andrew Woodward
> >
> > Mirantis
> >
> > Fuel Community Ambassador
> >
> > Ceph Community
> >
> >
> >
>  __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > <
> http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc 

Re: [openstack-dev] [fuel][ConfigDB] Separating node and cluster serialized data

[Bogdan Dobrelya]

On 04/01/2016 10:41 AM, Oleg Gelbukh wrote:
> Andrew,
> 
> This is an excellent idea. It is apparently more efficient and
> error-proof to make the split not by the resulted data but at the time
> it is actually generated. We will play with this idea a little bit, and
> will come up with design proposal shortly.
> 
> Meanwhile, please be informed that we already started testing the
> solution based on the node-level data exposed via ConfigDB API extension
> for Nailgun [1] [2]. I will keep you updated on our progress in that area.

I strongly believe that nodes must only consume data, not provide one.
And the data must be collected from its sources, which is Nailgun API
extensions, like Andrew described.

> 
> [1] Specification for Nailgun API for serialized facts
> 
> [2] Spec for upload of deployment configuration to ConfigDB API
> 
> 
> --
> Best regards,
> Oleg Gelbukh
> 
> On Thu, Mar 31, 2016 at 11:19 PM, Andrew Woodward  > wrote:
> 
> One of the problems we've faced with trying to plug-in ConfigDB is
> trying to separate the cluster attributes from the node attributes
> in the serialized output (ie astute.yaml)
> 
> I started talking with Alex S about how we could separate them after
> astute.yaml is prepared trying to ensure which was which we came
> back uncertain that the results would be accurate.
> 
> So I figured I'd go back to the source and see if there was a way to
> know which keys belonged where. It turns out that we could solve the
> problem in a simpler and more precise way than cutting them back
> apart later.
> 
> Looking over the deployment_serializers.py [1] the serialized data
> follows a simple work flow
> 
> iterate over every node in cluster
>   if node is customized:
> serialized_data = node.replaced_deployment_data
>   else:
> serialized_data = dict_merge(
>   serialize_node(node),
>   get_common_attrs(cluster))
> 
> Taking this into mind, we can simply construct an extension to
> expose these as an APIs so that we can consume them as a task in the
> deployment graph.
> 
> Cluster:
> We can simply expose
> DeploymentMultinodeSerializer().get_common_attrs(cluster)
> 
> This would then be plumbed to the cluster level in ConfigDB
> 
> Node:
> if a Node has customized data, then we can return that at the node
> level, this continues to work at the same as native since it most
> likely has Cluster merged into it.
> 
> otherwise we can return the serialized node with whichever of the
> first 'role' the node has
> 
> We would expose DeploymentMultinodeSerializer().serialize_node(node,
> objects.Node.all_roles(node)[0])
> 
> for our usage, we don't need to worry about the normal node role
> combination as the data only influences 'role' and 'fail_if_error'
> attributes, both are not consumed in the library.
> 
> 
> https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py#L93-L121
> -- 
> 
> --
> 
> Andrew Woodward
> 
> Mirantis
> 
> Fuel Community Ambassador
> 
> Ceph Community
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel][ConfigDB] Separating node and cluster serialized data

[Oleg Gelbukh]

Andrew,

This is an excellent idea. It is apparently more efficient and error-proof
to make the split not by the resulted data but at the time it is actually
generated. We will play with this idea a little bit, and will come up with
design proposal shortly.

Meanwhile, please be informed that we already started testing the solution
based on the node-level data exposed via ConfigDB API extension for Nailgun
[1] [2]. I will keep you updated on our progress in that area.

[1] Specification for Nailgun API for serialized facts

[2] Spec for upload of deployment configuration to ConfigDB API


--
Best regards,
Oleg Gelbukh

On Thu, Mar 31, 2016 at 11:19 PM, Andrew Woodward  wrote:

> One of the problems we've faced with trying to plug-in ConfigDB is trying
> to separate the cluster attributes from the node attributes in the
> serialized output (ie astute.yaml)
>
> I started talking with Alex S about how we could separate them after
> astute.yaml is prepared trying to ensure which was which we came back
> uncertain that the results would be accurate.
>
> So I figured I'd go back to the source and see if there was a way to know
> which keys belonged where. It turns out that we could solve the problem in
> a simpler and more precise way than cutting them back apart later.
>
> Looking over the deployment_serializers.py [1] the serialized data follows
> a simple work flow
>
> iterate over every node in cluster
>   if node is customized:
> serialized_data = node.replaced_deployment_data
>   else:
> serialized_data = dict_merge(
>   serialize_node(node),
>   get_common_attrs(cluster))
>
> Taking this into mind, we can simply construct an extension to expose
> these as an APIs so that we can consume them as a task in the deployment
> graph.
>
> Cluster:
> We can simply expose
> DeploymentMultinodeSerializer().get_common_attrs(cluster)
>
> This would then be plumbed to the cluster level in ConfigDB
>
> Node:
> if a Node has customized data, then we can return that at the node level,
> this continues to work at the same as native since it most likely has
> Cluster merged into it.
>
> otherwise we can return the serialized node with whichever of the first
> 'role' the node has
>
> We would expose DeploymentMultinodeSerializer().serialize_node(node,
> objects.Node.all_roles(node)[0])
>
> for our usage, we don't need to worry about the normal node role
> combination as the data only influences 'role' and 'fail_if_error'
> attributes, both are not consumed in the library.
>
>
> https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py#L93-L121
> --
>
> --
>
> Andrew Woodward
>
> Mirantis
>
> Fuel Community Ambassador
>
> Ceph Community
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [fuel][ConfigDB] Separating node and cluster serialized data

[Andrew Woodward]

One of the problems we've faced with trying to plug-in ConfigDB is trying
to separate the cluster attributes from the node attributes in the
serialized output (ie astute.yaml)

I started talking with Alex S about how we could separate them after
astute.yaml is prepared trying to ensure which was which we came back
uncertain that the results would be accurate.

So I figured I'd go back to the source and see if there was a way to know
which keys belonged where. It turns out that we could solve the problem in
a simpler and more precise way than cutting them back apart later.

Looking over the deployment_serializers.py [1] the serialized data follows
a simple work flow

iterate over every node in cluster
  if node is customized:
serialized_data = node.replaced_deployment_data
  else:
serialized_data = dict_merge(
  serialize_node(node),
  get_common_attrs(cluster))

Taking this into mind, we can simply construct an extension to expose these
as an APIs so that we can consume them as a task in the deployment graph.

Cluster:
We can simply expose
DeploymentMultinodeSerializer().get_common_attrs(cluster)

This would then be plumbed to the cluster level in ConfigDB

Node:
if a Node has customized data, then we can return that at the node level,
this continues to work at the same as native since it most likely has
Cluster merged into it.

otherwise we can return the serialized node with whichever of the first
'role' the node has

We would expose DeploymentMultinodeSerializer().serialize_node(node,
objects.Node.all_roles(node)[0])

for our usage, we don't need to worry about the normal node role
combination as the data only influences 'role' and 'fail_if_error'
attributes, both are not consumed in the library.

https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_serializers.py#L93-L121
-- 

--

Andrew Woodward

Mirantis

Fuel Community Ambassador

Ceph Community
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev