Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Bogdan Dobrelya]

On 22.12.2015 21:12, Yuriy Taraday wrote:
> Hello, everybody.
> 
> It's a week old thread and I should've jumped in earlier. Better late
> than never.
> 
> On Wed, Dec 16, 2015 at 2:04 AM Dmitriy Shulyak  > wrote:
> 
> Hello folks,
> 
> This topic is about configuration storage which will connect data
> sources (nailgun/bareon/others) and orchestration. And right now we
> are developing two projects that will overlap a bit.
> 
> I understand there is not enough context to dive into this thread
> right away, but i will appreciate if those people, who participated
> in design, will add their opinions/clarifications on this matter.
> 
> 
> Let's try to add more context here. I see a lot of confusion around this
> matter. I think most of it comes from not having a single complete
> source of data about both approaches. I'll try to summarize the problem
> and outline proposed solutions as well as state of implementation for those.
> 
> == The problems. ==
> 
> Currently we have 2 main problems in question:
> 1. How to store data in Fuel so that it can come from different sources
> and be consumed by different pieces of software.
> 2. How to integrate Solar with Fuel and allow it to consume data
> provided by Nailgun (currently) or whatever else (if we get #1 implemented).

IIUC, that exposes Config Service to Solar as a "proxy" data source
among with other external sources like Nailgun, LDAP, IPAM and so on.
But there is support of external data sources on the Solar radar as
well. Developing "data resources" with "data processors" in Solar in
parallel with the Config Service doing literally the same -but as a
proxy data access layer- would bring major duplication of efforts, which
is not good. Instead, we should re-think of a "flat model" with no data
proxies and using Solar framework from the very beginning, IMHO.

I suggest to address this issue ASAP.

> 
> I was assigned (driven, actually) to look at the problem #1, and so with
> a start of a number of ideas from Oleg and others from my team and after
> some discussion with other people involved in Fuel architecture, I've
> finalized the scope and outlined architecture of the service that we
> would need to solve it in [0]. I didn't mean to step on anyone's toes,
> but later I was shown that the similar issue is being solved by SolarDB
> initiative that is being developed in the scope of integration between
> Solar and Fuel.
> 
> == The solutions. ==
> 
> = Config Service =
> 
> (because ConfigDB became an overused and ill-defined term)
> 
> Config Service is thought as a way to link Nailgun (and other sources in
> the future) to deployment tasks (and other consumers in the future) in a
> consistent and verifiable way. Someone lets the service know about the
> structure of the data that components provide and the structure of the
> data that other components consume thus declaring internal data flow
> between sources and consumers of data. Then for every environment
> Nailgun (for now - it can be other service, it can be changed to pull
> model later) feeds all necessary data into the service, triggers
> deployment tasks that consume data from the service the way that is more
> suitable for them. If we need to feed this data into some external
> service (Puppet master, for example), we're free to do so as long as we
> define data structure that a consumer expects.
> 
> = SolarDB =
> 
> (mainly based on [1] and presentations seen earlier, please correct me
> if smth wrong)
> 
> SolarDB includes active component: Data Processors (DPs). DPs fetch data
> from wherever they're intended to (Nailgun for starters, any other
> source in the future) and store them as Solar's Data Resorces (DRs). DRs
> are then translated to concrete data for other Solar's Resources
> (Executable Resources, ERs), this data is preprocessed by Policy Engine
> and converted to a set of calls to mutators that change ERs data in
> Solar's internal database in a way that lets Solar decide what should be
> done to change actual state of the environment.
> 
> == State of implementations ==
> 
> = Config Service =
> 
> I plan to show a PoC for Config Service integration before the end of
> this year. Coding of the service itself is almost at finish line at [2],
> integration with Nailgun and Astute/Puppet will take most of the
> remaining time.
> 
> = SolarDB =
> 
> PoC with Data Processors and Data Resources happened with simple cluster
> architecture (I don't know the date here). Policy Engine is in early
> stages of development.
> 
> == Main differences ==
> 
> I'll try to list main differences along with what seems for me pros and
> cons for both sides (major points taken from original Dmitriy's email).
> 
> 1. Config Service (CS) is initially planned as a passive element of
> integration while SolarDB (SD) have DPs that actively fetch data from
> the sources.
> 
> CS+: simpler implementation, can get PoC 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Yuriy Taraday]

Hello, everybody.

It's a week old thread and I should've jumped in earlier. Better late than
never.

On Wed, Dec 16, 2015 at 2:04 AM Dmitriy Shulyak 
wrote:

> Hello folks,
>
> This topic is about configuration storage which will connect data sources
> (nailgun/bareon/others) and orchestration. And right now we are developing
> two projects that will overlap a bit.
>
> I understand there is not enough context to dive into this thread right
> away, but i will appreciate if those people, who participated in design,
> will add their opinions/clarifications on this matter.
>

Let's try to add more context here. I see a lot of confusion around this
matter. I think most of it comes from not having a single complete source
of data about both approaches. I'll try to summarize the problem and
outline proposed solutions as well as state of implementation for those.

== The problems. ==

Currently we have 2 main problems in question:
1. How to store data in Fuel so that it can come from different sources and
be consumed by different pieces of software.
2. How to integrate Solar with Fuel and allow it to consume data provided
by Nailgun (currently) or whatever else (if we get #1 implemented).

I was assigned (driven, actually) to look at the problem #1, and so with a
start of a number of ideas from Oleg and others from my team and after some
discussion with other people involved in Fuel architecture, I've finalized
the scope and outlined architecture of the service that we would need to
solve it in [0]. I didn't mean to step on anyone's toes, but later I was
shown that the similar issue is being solved by SolarDB initiative that is
being developed in the scope of integration between Solar and Fuel.

== The solutions. ==

= Config Service =

(because ConfigDB became an overused and ill-defined term)

Config Service is thought as a way to link Nailgun (and other sources in
the future) to deployment tasks (and other consumers in the future) in a
consistent and verifiable way. Someone lets the service know about the
structure of the data that components provide and the structure of the data
that other components consume thus declaring internal data flow between
sources and consumers of data. Then for every environment Nailgun (for now
- it can be other service, it can be changed to pull model later) feeds all
necessary data into the service, triggers deployment tasks that consume
data from the service the way that is more suitable for them. If we need to
feed this data into some external service (Puppet master, for example),
we're free to do so as long as we define data structure that a consumer
expects.

= SolarDB =

(mainly based on [1] and presentations seen earlier, please correct me if
smth wrong)

SolarDB includes active component: Data Processors (DPs). DPs fetch data
from wherever they're intended to (Nailgun for starters, any other source
in the future) and store them as Solar's Data Resorces (DRs). DRs are then
translated to concrete data for other Solar's Resources (Executable
Resources, ERs), this data is preprocessed by Policy Engine and converted
to a set of calls to mutators that change ERs data in Solar's internal
database in a way that lets Solar decide what should be done to change
actual state of the environment.

== State of implementations ==

= Config Service =

I plan to show a PoC for Config Service integration before the end of this
year. Coding of the service itself is almost at finish line at [2],
integration with Nailgun and Astute/Puppet will take most of the remaining
time.

= SolarDB =

PoC with Data Processors and Data Resources happened with simple cluster
architecture (I don't know the date here). Policy Engine is in early stages
of development.

== Main differences ==

I'll try to list main differences along with what seems for me pros and
cons for both sides (major points taken from original Dmitriy's email).

1. Config Service (CS) is initially planned as a passive element of
integration while SolarDB (SD) have DPs that actively fetch data from the
sources.

CS+: simpler implementation, can get PoC done fast
CS+: easier to integrate with current Fuel architecture (actors remain
actors there)
CS+: can be easily modified to add active components to the service (pull
model)
CS-: brings in another component into the stack
CS-: requires other components to be changed to push data into the service
SD+: doesn't require any changes in other components
SD-: requires Solar to store fetched data
SD-: brings in another component into the stack

Overall I think that this point is not crucial here: both approaches can be
converted back and forth without much effort in both solutions.

2. Config Service is designed as an independent service, SolarDB is tight
to Solar infrastructure.

CS+: defines clear interface between components
CS+: doesn't require Solar at all, so we can land and use it independently
CS-: duplicates data between Config Service and Solar
SD+: integrated 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Evgeniy L]

Hi Oleg,

I understand the concern, but in case of integration specifically with
Solar,
I don't see any reasons to add ConfigDB, because Solar by itself is a
ConfigDB.
At the same time I would agree that there might be a case, when user uses
Zookeeper/Puppet Master/CMDB as a data store, in this case we should store
the data directly in those services, without keeping them in yet another
storage.

So the flow will look like this:
Components ->
Data get polled by data processors ->
| Solar data processor puts the data into Solar in its format
| Zookeeper data processor puts the data into Zookeeper in its format
| Custom CMDB data processor puts the data into CMDB in its own format

Thanks,

On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh  wrote:

> Hi,
>
> The idea behind configdb is that it is independent component that defines
> data flows between other components of the system. It has no knowledge
> about those components or specifics of their data. Data formats are defined
> by components themselves via schemas/templates and can be changed at any
> time (i.e. don't require code changes).
>
> Important 'pro' having ConfigDB separate from Solar is that it will
> simplify transition from current Fuel architecture by breaking it into more
> definite stages and reducing the number of components Solar have to be
> integrated with.
>
> --
> Best regards,
> Oleg Gelbukh
>
> On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:
>
>> Hi Dmitry,
>>
>> I also don't think that we should duplicate the data in configdb,
>> because in this case there will be +2 additional interfaces which
>> will require to covert the data into configdb and after that from
>> configdb to Solar, which seems redundant overhead.
>>
>> But we should be able to put the data directly to user's
>> CMDB/ZooKeeper/Puppet Master/etc.
>>
>> Thanks,
>>
>> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak 
>> wrote:
>>
>>> Hello folks,
>>>
>>> This topic is about configuration storage which will connect data
>>> sources (nailgun/bareon/others) and orchestration. And right now we are
>>> developing two projects that will overlap a bit.
>>>
>>> I understand there is not enough context to dive into this thread right
>>> away, but i will appreciate if those people, who participated in design,
>>> will add their opinions/clarifications on this matter.
>>>
>>> Main disagreements
>>> ---
>>> 1. configdb should be passive, writing to configdb is someone else
>>> responsibility
>>> + simpler implementation, easier to use
>>> - we will need another component that will do writing, or split this
>>> responsibility somehow
>>>
>>> 2. can be used without other solar components
>>> + clear inteface between solar components and storage layer
>>> - additional work required to design/refactor communication layer
>>> between modules in solar
>>> - some data will be duplicated between solar orchestrator layer and
>>> configdb
>>>
>>> 3. templates for output
>>> technical detail, can be added on top of solardb if required
>>>
>>> Similar functionality
>>> --
>>> 1. Hierachical storage
>>> 2. Versioning of changes
>>> 3. Possibility to overwrite config values
>>> 4. Schema for inputs
>>>
>>> Overall it seems that we share same goals for both services,
>>> the difference lies in organizational and technical implementation
>>> details.
>>>
>>> Possible solutions
>>> 
>>> 1. develop configdb and solar with duplicated functionality
>>> - at least 2 additional components will be added to the picture,
>>> one is configdb, another one will need to sync data between configdb and
>>> solar
>>> - in some cases data in solar and configdb will be 100% duplicated
>>> - different teams will work on same functionality
>>> - integration of additional component for fuel will require integration
>>> with
>>> configdb and with solar
>>> + configdb will be independent from solar orchestration/other components
>>>
>>> 2. make service out of solardb, allign with configdb use cases
>>> + solardb will be independent from solar orchestration/other solar
>>> components
>>> + integration of fuel component will be easier than in 1st version
>>> + clarity about components responsibility and new architecture
>>> - redesign/refactoring communication between components in solar
>>>
>>> 3. do not use configdb/no extraction of solardb
>>> - inproc communication, which can lead to coupled components (not the
>>> case currently)
>>> + faster implementation (no major changes required for integration with
>>> fuel)
>>> + clarity about components responsibility and new architecture
>>>
>>> Summary
>>> -
>>> For solar it makes no difference where data will come from: configdb or
>>> data sources, but in overall fuel architecture it will lead to
>>> significant
>>> complexity increase.
>>> It would be the best to follow 2nd path, because in long term we don't
>>> want 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Oleg Gelbukh]

The problem with this approach is that we can't manage the interfaces
between components without changing the code of 2+ components (i.e. one
that provides the data and all that consume it).

I also don't like polling model for data processors. In my understanding,
components should push their changes through the pipeline. Although this is
pure implementation detail and is not really important ATM.

The point is that for Solar integration, we still need integration points,
and the less of them we have, the more simple the transition is going to
be..

--
Best regards,
Oleg Gelbukh

On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L  wrote:

> Hi Oleg,
>
> I understand the concern, but in case of integration specifically with
> Solar,
> I don't see any reasons to add ConfigDB, because Solar by itself is a
> ConfigDB.
> At the same time I would agree that there might be a case, when user uses
> Zookeeper/Puppet Master/CMDB as a data store, in this case we should store
> the data directly in those services, without keeping them in yet another
> storage.
>
> So the flow will look like this:
> Components ->
> Data get polled by data processors ->
> | Solar data processor puts the data into Solar in its format
> | Zookeeper data processor puts the data into Zookeeper in its format
> | Custom CMDB data processor puts the data into CMDB in its own format
>
> Thanks,
>
> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh 
> wrote:
>
>> Hi,
>>
>> The idea behind configdb is that it is independent component that defines
>> data flows between other components of the system. It has no knowledge
>> about those components or specifics of their data. Data formats are defined
>> by components themselves via schemas/templates and can be changed at any
>> time (i.e. don't require code changes).
>>
>> Important 'pro' having ConfigDB separate from Solar is that it will
>> simplify transition from current Fuel architecture by breaking it into more
>> definite stages and reducing the number of components Solar have to be
>> integrated with.
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:
>>
>>> Hi Dmitry,
>>>
>>> I also don't think that we should duplicate the data in configdb,
>>> because in this case there will be +2 additional interfaces which
>>> will require to covert the data into configdb and after that from
>>> configdb to Solar, which seems redundant overhead.
>>>
>>> But we should be able to put the data directly to user's
>>> CMDB/ZooKeeper/Puppet Master/etc.
>>>
>>> Thanks,
>>>
>>> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak 
>>> wrote:
>>>
 Hello folks,

 This topic is about configuration storage which will connect data
 sources (nailgun/bareon/others) and orchestration. And right now we are
 developing two projects that will overlap a bit.

 I understand there is not enough context to dive into this thread right
 away, but i will appreciate if those people, who participated in design,
 will add their opinions/clarifications on this matter.

 Main disagreements
 ---
 1. configdb should be passive, writing to configdb is someone else
 responsibility
 + simpler implementation, easier to use
 - we will need another component that will do writing, or split this
 responsibility somehow

 2. can be used without other solar components
 + clear inteface between solar components and storage layer
 - additional work required to design/refactor communication layer
 between modules in solar
 - some data will be duplicated between solar orchestrator layer and
 configdb

 3. templates for output
 technical detail, can be added on top of solardb if required

 Similar functionality
 --
 1. Hierachical storage
 2. Versioning of changes
 3. Possibility to overwrite config values
 4. Schema for inputs

 Overall it seems that we share same goals for both services,
 the difference lies in organizational and technical implementation
 details.

 Possible solutions
 
 1. develop configdb and solar with duplicated functionality
 - at least 2 additional components will be added to the picture,
 one is configdb, another one will need to sync data between configdb
 and solar
 - in some cases data in solar and configdb will be 100% duplicated
 - different teams will work on same functionality
 - integration of additional component for fuel will require integration
 with
 configdb and with solar
 + configdb will be independent from solar orchestration/other components

 2. make service out of solardb, allign with configdb use cases
 + solardb will be independent from solar orchestration/other solar
 components
 + integration of 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Evgeniy L]

Comments inline.

On Mon, Dec 21, 2015 at 1:42 PM, Oleg Gelbukh  wrote:

> The problem with this approach is that we can't manage the interfaces
> between components without changing the code of 2+ components (i.e. one
> that provides the data and all that consume it).
>

I'm not sure if understand you correctly, none of these approaches
require to change the components, we will have change a single
place, which is specific data processor.


>
> I also don't like polling model for data processors. In my understanding,
> components should push their changes through the pipeline. Although this is
> pure implementation detail and is not really important ATM.
>

We don't have any other choice, we don't control components, configuration
components are not going to implement Fuel specific logic, so Fuel has
to pool the data.


>
> The point is that for Solar integration, we still need integration points,
> and the less of them we have, the more simple the transition is going to
> be..
>

As described above, there will be a single integration point, data
processor.

>
> --
> Best regards,
> Oleg Gelbukh
>
> On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L  wrote:
>
>> Hi Oleg,
>>
>> I understand the concern, but in case of integration specifically with
>> Solar,
>> I don't see any reasons to add ConfigDB, because Solar by itself is a
>> ConfigDB.
>> At the same time I would agree that there might be a case, when user uses
>> Zookeeper/Puppet Master/CMDB as a data store, in this case we should store
>> the data directly in those services, without keeping them in yet another
>> storage.
>>
>> So the flow will look like this:
>> Components ->
>> Data get polled by data processors ->
>> | Solar data processor puts the data into Solar in its format
>> | Zookeeper data processor puts the data into Zookeeper in its format
>> | Custom CMDB data processor puts the data into CMDB in its own format
>>
>> Thanks,
>>
>> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh 
>> wrote:
>>
>>> Hi,
>>>
>>> The idea behind configdb is that it is independent component that
>>> defines data flows between other components of the system. It has no
>>> knowledge about those components or specifics of their data. Data formats
>>> are defined by components themselves via schemas/templates and can be
>>> changed at any time (i.e. don't require code changes).
>>>
>>> Important 'pro' having ConfigDB separate from Solar is that it will
>>> simplify transition from current Fuel architecture by breaking it into more
>>> definite stages and reducing the number of components Solar have to be
>>> integrated with.
>>>
>>> --
>>> Best regards,
>>> Oleg Gelbukh
>>>
>>> On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:
>>>
 Hi Dmitry,

 I also don't think that we should duplicate the data in configdb,
 because in this case there will be +2 additional interfaces which
 will require to covert the data into configdb and after that from
 configdb to Solar, which seems redundant overhead.

 But we should be able to put the data directly to user's
 CMDB/ZooKeeper/Puppet Master/etc.

 Thanks,

 On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak  wrote:

> Hello folks,
>
> This topic is about configuration storage which will connect data
> sources (nailgun/bareon/others) and orchestration. And right now we are
> developing two projects that will overlap a bit.
>
> I understand there is not enough context to dive into this thread
> right away, but i will appreciate if those people, who participated in
> design, will add their opinions/clarifications on this matter.
>
> Main disagreements
> ---
> 1. configdb should be passive, writing to configdb is someone else
> responsibility
> + simpler implementation, easier to use
> - we will need another component that will do writing, or split this
> responsibility somehow
>
> 2. can be used without other solar components
> + clear inteface between solar components and storage layer
> - additional work required to design/refactor communication layer
> between modules in solar
> - some data will be duplicated between solar orchestrator layer and
> configdb
>
> 3. templates for output
> technical detail, can be added on top of solardb if required
>
> Similar functionality
> --
> 1. Hierachical storage
> 2. Versioning of changes
> 3. Possibility to overwrite config values
> 4. Schema for inputs
>
> Overall it seems that we share same goals for both services,
> the difference lies in organizational and technical implementation
> details.
>
> Possible solutions
> 
> 1. develop configdb and solar with duplicated functionality
> - at 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

I definitely agree with what Evgeniy said.

@Oleg could you make a step-by-step how do you imagine this
integration (with separate ConfigDB) ? To me this adds at least one
component / integration point.


> The point is that for Solar integration, we still need integration points, 
> and the less of them we have, the more simple the transition is going to be..

With DR+DP you will have exactly one integration point.

> I also don't like polling model for data processors. In my understanding, 
> components should push their changes through the pipeline. Although this is 
> pure implementation detail and is not really important ATM.

Push is problematic because:
1) if you don't own all components then you're in troubles
2) you never know how valid are data of component (you could add some
time based markers, but still it's not enough), you basically loose
control about data. With pull model you know more about context etc.

On Mon, Dec 21, 2015 at 11:49 AM, Evgeniy L  wrote:
> Comments inline.
>
> On Mon, Dec 21, 2015 at 1:42 PM, Oleg Gelbukh  wrote:
>>
>> The problem with this approach is that we can't manage the interfaces
>> between components without changing the code of 2+ components (i.e. one that
>> provides the data and all that consume it).
>
>
> I'm not sure if understand you correctly, none of these approaches
> require to change the components, we will have change a single
> place, which is specific data processor.
>
>>
>>
>> I also don't like polling model for data processors. In my understanding,
>> components should push their changes through the pipeline. Although this is
>> pure implementation detail and is not really important ATM.
>
>
> We don't have any other choice, we don't control components, configuration
> components are not going to implement Fuel specific logic, so Fuel has
> to pool the data.
>
>>
>>
>> The point is that for Solar integration, we still need integration points,
>> and the less of them we have, the more simple the transition is going to
>> be..
>
>
> As described above, there will be a single integration point, data
> processor.
>>
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L  wrote:
>>>
>>> Hi Oleg,
>>>
>>> I understand the concern, but in case of integration specifically with
>>> Solar,
>>> I don't see any reasons to add ConfigDB, because Solar by itself is a
>>> ConfigDB.
>>> At the same time I would agree that there might be a case, when user uses
>>> Zookeeper/Puppet Master/CMDB as a data store, in this case we should
>>> store
>>> the data directly in those services, without keeping them in yet another
>>> storage.
>>>
>>> So the flow will look like this:
>>> Components ->
>>> Data get polled by data processors ->
>>> | Solar data processor puts the data into Solar in its format
>>> | Zookeeper data processor puts the data into Zookeeper in its format
>>> | Custom CMDB data processor puts the data into CMDB in its own format
>>>
>>> Thanks,
>>>
>>> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh 
>>> wrote:

 Hi,

 The idea behind configdb is that it is independent component that
 defines data flows between other components of the system. It has no
 knowledge about those components or specifics of their data. Data formats
 are defined by components themselves via schemas/templates and can be
 changed at any time (i.e. don't require code changes).

 Important 'pro' having ConfigDB separate from Solar is that it will
 simplify transition from current Fuel architecture by breaking it into more
 definite stages and reducing the number of components Solar have to be
 integrated with.

 --
 Best regards,
 Oleg Gelbukh

 On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:
>
> Hi Dmitry,
>
> I also don't think that we should duplicate the data in configdb,
> because in this case there will be +2 additional interfaces which
> will require to covert the data into configdb and after that from
> configdb to Solar, which seems redundant overhead.
>
> But we should be able to put the data directly to user's
> CMDB/ZooKeeper/Puppet Master/etc.
>
> Thanks,
>
> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak
>  wrote:
>>
>> Hello folks,
>>
>> This topic is about configuration storage which will connect data
>> sources (nailgun/bareon/others) and orchestration. And right now we are
>> developing two projects that will overlap a bit.
>>
>> I understand there is not enough context to dive into this thread
>> right away, but i will appreciate if those people, who participated in
>> design, will add their opinions/clarifications on this matter.
>>
>> Main disagreements
>> ---
>> 1. configdb should be 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Bogdan Dobrelya]

On 21.12.2015 11:42, Oleg Gelbukh wrote:
> The problem with this approach is that we can't manage the interfaces
> between components without changing the code of 2+ components (i.e. one
> that provides the data and all that consume it).

What code is supposed to be changed in an example LDAP data source?
Otherwise, the only place I see shall require code changes is a data
processor for it, if one may want to support new (or very old) version
of the ldap with another database schema.

> 
> I also don't like polling model for data processors. In my
> understanding, components should push their changes through the
> pipeline. Although this is pure implementation detail and is not really
> important ATM.

What do you think the example LDAP data source shall push? IIUC, for
most auth cases, it shall be only pulled and processed.

> 
> The point is that for Solar integration, we still need integration
> points, and the less of them we have, the more simple the transition is
> going to be..

Data processors may be the only integration point, as they are supposed
to "know" how-to get and how-to shape data from sources

> 
> --
> Best regards,
> Oleg Gelbukh
> 
> On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L  > wrote:
> 
> Hi Oleg,
> 
> I understand the concern, but in case of integration specifically
> with Solar,
> I don't see any reasons to add ConfigDB, because Solar by itself is
> a ConfigDB.
> At the same time I would agree that there might be a case, when user
> uses
> Zookeeper/Puppet Master/CMDB as a data store, in this case we should
> store
> the data directly in those services, without keeping them in yet
> another storage.
> 
> So the flow will look like this:
> Components ->
> Data get polled by data processors ->
> | Solar data processor puts the data into Solar in its format
> | Zookeeper data processor puts the data into Zookeeper in its format
> | Custom CMDB data processor puts the data into CMDB in its own format
> 
> Thanks,
> 
> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh  > wrote:
> 
> Hi,
> 
> The idea behind configdb is that it is independent component
> that defines data flows between other components of the system.
> It has no knowledge about those components or specifics of their
> data. Data formats are defined by components themselves via
> schemas/templates and can be changed at any time (i.e. don't
> require code changes).
> 
> Important 'pro' having ConfigDB separate from Solar is that it
> will simplify transition from current Fuel architecture by
> breaking it into more definite stages and reducing the number of
> components Solar have to be integrated with.
> 
> --
> Best regards,
> Oleg Gelbukh
> 
> On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  > wrote:
> 
> Hi Dmitry,
> 
> I also don't think that we should duplicate the data in
> configdb,
> because in this case there will be +2 additional interfaces
> which
> will require to covert the data into configdb and after that
> from
> configdb to Solar, which seems redundant overhead.
> 
> But we should be able to put the data directly to user's
> CMDB/ZooKeeper/Puppet Master/etc.
> 
> Thanks,
> 
> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak
> > wrote:
> 
> Hello folks,
> 
> This topic is about configuration storage which will
> connect data sources (nailgun/bareon/others) and
> orchestration. And right now we are developing two
> projects that will overlap a bit.
> 
> I understand there is not enough context to dive into
> this thread right away, but i will appreciate if those
> people, who participated in design, will add their
> opinions/clarifications on this matter.
> 
> Main disagreements
> ---
> 1. configdb should be passive, writing to configdb is
> someone else responsibility
> + simpler implementation, easier to use
> - we will need another component that will do writing,
> or split this responsibility somehow
> 
> 2. can be used without other solar components
> + clear inteface between solar components and storage layer
> - additional work required to design/refactor
> communication layer between modules in solar
>   

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Oleg Gelbukh]

Hi,

The idea behind configdb is that it is independent component that defines
data flows between other components of the system. It has no knowledge
about those components or specifics of their data. Data formats are defined
by components themselves via schemas/templates and can be changed at any
time (i.e. don't require code changes).

Important 'pro' having ConfigDB separate from Solar is that it will
simplify transition from current Fuel architecture by breaking it into more
definite stages and reducing the number of components Solar have to be
integrated with.

--
Best regards,
Oleg Gelbukh

On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:

> Hi Dmitry,
>
> I also don't think that we should duplicate the data in configdb,
> because in this case there will be +2 additional interfaces which
> will require to covert the data into configdb and after that from
> configdb to Solar, which seems redundant overhead.
>
> But we should be able to put the data directly to user's
> CMDB/ZooKeeper/Puppet Master/etc.
>
> Thanks,
>
> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak 
> wrote:
>
>> Hello folks,
>>
>> This topic is about configuration storage which will connect data sources
>> (nailgun/bareon/others) and orchestration. And right now we are developing
>> two projects that will overlap a bit.
>>
>> I understand there is not enough context to dive into this thread right
>> away, but i will appreciate if those people, who participated in design,
>> will add their opinions/clarifications on this matter.
>>
>> Main disagreements
>> ---
>> 1. configdb should be passive, writing to configdb is someone else
>> responsibility
>> + simpler implementation, easier to use
>> - we will need another component that will do writing, or split this
>> responsibility somehow
>>
>> 2. can be used without other solar components
>> + clear inteface between solar components and storage layer
>> - additional work required to design/refactor communication layer between
>> modules in solar
>> - some data will be duplicated between solar orchestrator layer and
>> configdb
>>
>> 3. templates for output
>> technical detail, can be added on top of solardb if required
>>
>> Similar functionality
>> --
>> 1. Hierachical storage
>> 2. Versioning of changes
>> 3. Possibility to overwrite config values
>> 4. Schema for inputs
>>
>> Overall it seems that we share same goals for both services,
>> the difference lies in organizational and technical implementation
>> details.
>>
>> Possible solutions
>> 
>> 1. develop configdb and solar with duplicated functionality
>> - at least 2 additional components will be added to the picture,
>> one is configdb, another one will need to sync data between configdb and
>> solar
>> - in some cases data in solar and configdb will be 100% duplicated
>> - different teams will work on same functionality
>> - integration of additional component for fuel will require integration
>> with
>> configdb and with solar
>> + configdb will be independent from solar orchestration/other components
>>
>> 2. make service out of solardb, allign with configdb use cases
>> + solardb will be independent from solar orchestration/other solar
>> components
>> + integration of fuel component will be easier than in 1st version
>> + clarity about components responsibility and new architecture
>> - redesign/refactoring communication between components in solar
>>
>> 3. do not use configdb/no extraction of solardb
>> - inproc communication, which can lead to coupled components (not the
>> case currently)
>> + faster implementation (no major changes required for integration with
>> fuel)
>> + clarity about components responsibility and new architecture
>>
>> Summary
>> -
>> For solar it makes no difference where data will come from: configdb or
>> data sources, but in overall fuel architecture it will lead to significant
>> complexity increase.
>> It would be the best to follow 2nd path, because in long term we don't
>> want tightly coupled components, but in nearest future we need to
>> concentrate
>> on:
>> - integration with fuel
>> - implementing policy engine
>> - polishing solar components
>> This is why i am not sure that we can spend time on 2nd path right now,
>> or even before 9.0.
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Evgeniy L]

Hi Dmitry,

I also don't think that we should duplicate the data in configdb,
because in this case there will be +2 additional interfaces which
will require to covert the data into configdb and after that from
configdb to Solar, which seems redundant overhead.

But we should be able to put the data directly to user's
CMDB/ZooKeeper/Puppet Master/etc.

Thanks,

On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak 
wrote:

> Hello folks,
>
> This topic is about configuration storage which will connect data sources
> (nailgun/bareon/others) and orchestration. And right now we are developing
> two projects that will overlap a bit.
>
> I understand there is not enough context to dive into this thread right
> away, but i will appreciate if those people, who participated in design,
> will add their opinions/clarifications on this matter.
>
> Main disagreements
> ---
> 1. configdb should be passive, writing to configdb is someone else
> responsibility
> + simpler implementation, easier to use
> - we will need another component that will do writing, or split this
> responsibility somehow
>
> 2. can be used without other solar components
> + clear inteface between solar components and storage layer
> - additional work required to design/refactor communication layer between
> modules in solar
> - some data will be duplicated between solar orchestrator layer and
> configdb
>
> 3. templates for output
> technical detail, can be added on top of solardb if required
>
> Similar functionality
> --
> 1. Hierachical storage
> 2. Versioning of changes
> 3. Possibility to overwrite config values
> 4. Schema for inputs
>
> Overall it seems that we share same goals for both services,
> the difference lies in organizational and technical implementation details.
>
> Possible solutions
> 
> 1. develop configdb and solar with duplicated functionality
> - at least 2 additional components will be added to the picture,
> one is configdb, another one will need to sync data between configdb and
> solar
> - in some cases data in solar and configdb will be 100% duplicated
> - different teams will work on same functionality
> - integration of additional component for fuel will require integration
> with
> configdb and with solar
> + configdb will be independent from solar orchestration/other components
>
> 2. make service out of solardb, allign with configdb use cases
> + solardb will be independent from solar orchestration/other solar
> components
> + integration of fuel component will be easier than in 1st version
> + clarity about components responsibility and new architecture
> - redesign/refactoring communication between components in solar
>
> 3. do not use configdb/no extraction of solardb
> - inproc communication, which can lead to coupled components (not the case
> currently)
> + faster implementation (no major changes required for integration with
> fuel)
> + clarity about components responsibility and new architecture
>
> Summary
> -
> For solar it makes no difference where data will come from: configdb or
> data sources, but in overall fuel architecture it will lead to significant
> complexity increase.
> It would be the best to follow 2nd path, because in long term we don't
> want tightly coupled components, but in nearest future we need to
> concentrate
> on:
> - integration with fuel
> - implementing policy engine
> - polishing solar components
> This is why i am not sure that we can spend time on 2nd path right now,
> or even before 9.0.
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

+1 for Lukasz concerns.

But if we really need operate with "solar resources database" as a kv
store, then we could implement service on top of it, It could be then
separate project, which would work as separate service. Would it fulfill
the requirements ? (we could implement it using some already existing
protocol).

On Wed, Dec 16, 2015 at 12:06 PM, Lukasz Oles  wrote:

> Hi Dima,
>
> On Wed, Dec 16, 2015 at 12:03 AM, Dmitriy Shulyak 
> wrote:
> > Hello folks,
> >
> > This topic is about configuration storage which will connect data sources
> > (nailgun/bareon/others) and orchestration. And right now we are
> developing
> > two projects that will overlap a bit.
> What 2 projects?
>
> >
> > I understand there is not enough context to dive into this thread right
> > away, but i will appreciate if those people, who participated in design,
> > will add their opinions/clarifications on this matter.
> >
> > Main disagreements
> > ---
> > 1. configdb should be passive, writing to configdb is someone else
> > responsibility
> > + simpler implementation, easier to use
> > - we will need another component that will do writing, or split this
> > responsibility somehow
> >
> > 2. can be used without other solar components
> > + clear inteface between solar components and storage layer
> > - additional work required to design/refactor communication layer between
> > modules in solar
> > - some data will be duplicated between solar orchestrator layer and
> configdb
> >
> > 3. templates for output
> > technical detail, can be added on top of solardb if required
> Who disagrees with who and about what? It's not clear for me. As far
> as I remember we agreed to not use configdb but to use solar "data
> resources". Data would be fetched using managers  in the way as you
> are doing now in f2s branch.
> What is the problem here?
>
> Regards
>
>
> > Similar functionality
> > --
> > 1. Hierachical storage
> > 2. Versioning of changes
> > 3. Possibility to overwrite config values
> > 4. Schema for inputs
> >
> > Overall it seems that we share same goals for both services,
> > the difference lies in organizational and technical implementation
> details.
> >
> > Possible solutions
> > 
> > 1. develop configdb and solar with duplicated functionality
> > - at least 2 additional components will be added to the picture,
> > one is configdb, another one will need to sync data between configdb and
> > solar
> > - in some cases data in solar and configdb will be 100% duplicated
> > - different teams will work on same functionality
> > - integration of additional component for fuel will require integration
> with
> > configdb and with solar
> > + configdb will be independent from solar orchestration/other components
> >
> > 2. make service out of solardb, allign with configdb use cases
> > + solardb will be independent from solar orchestration/other solar
> > components
> > + integration of fuel component will be easier than in 1st version
> > + clarity about components responsibility and new architecture
> > - redesign/refactoring communication between components in solar
> >
> > 3. do not use configdb/no extraction of solardb
> > - inproc communication, which can lead to coupled components (not the
> case
> > currently)
> > + faster implementation (no major changes required for integration with
> > fuel)
> > + clarity about components responsibility and new architecture
> >
> > Summary
> > -
> > For solar it makes no difference where data will come from: configdb or
> > data sources, but in overall fuel architecture it will lead to
> significant
> > complexity increase.
> > It would be the best to follow 2nd path, because in long term we don't
> want
> > tightly coupled components, but in nearest future we need to concentrate
> > on:
> > - integration with fuel
> > - implementing policy engine
> > - polishing solar components
> > This is why i am not sure that we can spend time on 2nd path right now,
> > or even before 9.0.
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Łukasz Oleś
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
--
Warm regards,
Jędrzej Nowak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Lukasz Oles]

Hi Dima,

On Wed, Dec 16, 2015 at 12:03 AM, Dmitriy Shulyak  wrote:
> Hello folks,
>
> This topic is about configuration storage which will connect data sources
> (nailgun/bareon/others) and orchestration. And right now we are developing
> two projects that will overlap a bit.
What 2 projects?

>
> I understand there is not enough context to dive into this thread right
> away, but i will appreciate if those people, who participated in design,
> will add their opinions/clarifications on this matter.
>
> Main disagreements
> ---
> 1. configdb should be passive, writing to configdb is someone else
> responsibility
> + simpler implementation, easier to use
> - we will need another component that will do writing, or split this
> responsibility somehow
>
> 2. can be used without other solar components
> + clear inteface between solar components and storage layer
> - additional work required to design/refactor communication layer between
> modules in solar
> - some data will be duplicated between solar orchestrator layer and configdb
>
> 3. templates for output
> technical detail, can be added on top of solardb if required
Who disagrees with who and about what? It's not clear for me. As far
as I remember we agreed to not use configdb but to use solar "data
resources". Data would be fetched using managers  in the way as you
are doing now in f2s branch.
What is the problem here?

Regards


> Similar functionality
> --
> 1. Hierachical storage
> 2. Versioning of changes
> 3. Possibility to overwrite config values
> 4. Schema for inputs
>
> Overall it seems that we share same goals for both services,
> the difference lies in organizational and technical implementation details.
>
> Possible solutions
> 
> 1. develop configdb and solar with duplicated functionality
> - at least 2 additional components will be added to the picture,
> one is configdb, another one will need to sync data between configdb and
> solar
> - in some cases data in solar and configdb will be 100% duplicated
> - different teams will work on same functionality
> - integration of additional component for fuel will require integration with
> configdb and with solar
> + configdb will be independent from solar orchestration/other components
>
> 2. make service out of solardb, allign with configdb use cases
> + solardb will be independent from solar orchestration/other solar
> components
> + integration of fuel component will be easier than in 1st version
> + clarity about components responsibility and new architecture
> - redesign/refactoring communication between components in solar
>
> 3. do not use configdb/no extraction of solardb
> - inproc communication, which can lead to coupled components (not the case
> currently)
> + faster implementation (no major changes required for integration with
> fuel)
> + clarity about components responsibility and new architecture
>
> Summary
> -
> For solar it makes no difference where data will come from: configdb or
> data sources, but in overall fuel architecture it will lead to significant
> complexity increase.
> It would be the best to follow 2nd path, because in long term we don't want
> tightly coupled components, but in nearest future we need to concentrate
> on:
> - integration with fuel
> - implementing policy engine
> - polishing solar components
> This is why i am not sure that we can spend time on 2nd path right now,
> or even before 9.0.
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Łukasz Oleś

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

On Wed, Dec 16, 2015 at 12:40 PM, Bogdan Dobrelya
 wrote:
> Data resources shall fill the configdb by results of the fetched data
> shaped by data processors aka serializers. The shaping process assumes
> applying of all versioning and schema transformations knowledge required
> to convert data from external sources into expected end state to make it
> consumable by the Policy Engine.
[..]
> Anyway, a
> solar resource just cannot be used without other solar components.

It depends what would be their usage, we could make them "like KV
store" with external access with some simple existing protocol (no
transactions, no bulk operations etc). Also it should be limited only
to DRs. Then we will have it's data, and PE would be responsible for
correct connections, DP - like in f2s - still could be possible/valid
in this case.
But as I said, it all depends on use cases.


-- 
--
Warm regards,
Jędrzej Nowak

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Dmitriy Shulyak]

Hello folks,

This topic is about configuration storage which will connect data sources
(nailgun/bareon/others) and orchestration. And right now we are developing
two projects that will overlap a bit.

I understand there is not enough context to dive into this thread right
away, but i will appreciate if those people, who participated in design,
will add their opinions/clarifications on this matter.

Main disagreements
---
1. configdb should be passive, writing to configdb is someone else
responsibility
+ simpler implementation, easier to use
- we will need another component that will do writing, or split this
responsibility somehow

2. can be used without other solar components
+ clear inteface between solar components and storage layer
- additional work required to design/refactor communication layer between
modules in solar
- some data will be duplicated between solar orchestrator layer and configdb

3. templates for output
technical detail, can be added on top of solardb if required

Similar functionality
--
1. Hierachical storage
2. Versioning of changes
3. Possibility to overwrite config values
4. Schema for inputs

Overall it seems that we share same goals for both services,
the difference lies in organizational and technical implementation details.

Possible solutions

1. develop configdb and solar with duplicated functionality
- at least 2 additional components will be added to the picture,
one is configdb, another one will need to sync data between configdb and
solar
- in some cases data in solar and configdb will be 100% duplicated
- different teams will work on same functionality
- integration of additional component for fuel will require integration with
configdb and with solar
+ configdb will be independent from solar orchestration/other components

2. make service out of solardb, allign with configdb use cases
+ solardb will be independent from solar orchestration/other solar
components
+ integration of fuel component will be easier than in 1st version
+ clarity about components responsibility and new architecture
- redesign/refactoring communication between components in solar

3. do not use configdb/no extraction of solardb
- inproc communication, which can lead to coupled components (not the case
currently)
+ faster implementation (no major changes required for integration with
fuel)
+ clarity about components responsibility and new architecture

Summary
-
For solar it makes no difference where data will come from: configdb or
data sources, but in overall fuel architecture it will lead to significant
complexity increase.
It would be the best to follow 2nd path, because in long term we don't want
tightly coupled components, but in nearest future we need to concentrate
on:
- integration with fuel
- implementing policy engine
- polishing solar components
This is why i am not sure that we can spend time on 2nd path right now,
or even before 9.0.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev