[openstack-dev] [Solar] Weekly update

[Jedrzej Nowak]

Hello,

This is weekly update from Solar project.

F2S:
- we're working on Class2 LCM demo using Fuel tasks

Solar itself:
- staging procedure supports now both implicit and explicit stages
- centos 7 patches merged (can be used instead of ubuntu)
- better pg pool implementation
- various bug fixes

Also, last Saturday I had a short presentation about Solar in Katowice.

--
Warm regards
Jedrzej Nowak

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Solar] Weekly update

[Jedrzej Nowak]

Hi,

Here is weekly update from Solar team

F2S:
- Cast uids to list and add a flag to make master and null optional
- Check if master is already created
- Every time when fuel_data is updated - insert anchors
- Add documentation for updates and make fuel env command idempotent

Solar itself:
- rework staging procedure to support both implicit and explicit stages
- vbox/qemu packer build added
- improvements for BAT transport
- fix testing examples using fuel-devops
- further packaging improvements


--
Warm regards,
Jedrzej Nowak

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Solar] Weekly update

[Jedrzej Nowak]

Hi,

Here is weekly update from Solar team

F2S:
- mapped all fuel-library tasks to solar resources
- removed resources from repo and generating them from installed
fuel-library version
- better UX of f2s usage
- sucessfully deployed controller+compute+cinder scenario
- working on running bvt and swarm tests
- activities on packaging

Solar itself:
- various UX improvements
- various bugs fixed
- packaging improvements



--
Warm regards,
Jedrzej Nowak

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel][Solar] Integration in 9.0

[Jedrzej Nowak]

As you may now we should integrate solar with fuel in 9.0. Integration in 9.0
will be very experimental and it will not affect fuel except: spec [](https://
review.openstack.org/283600)[https://review.openstack.org/283600](https://revi
ew.openstack.org/283600) and [https://review.openstack.org/#/c/284293/](https:
//review.openstack.org/#/c/284293/)

  

We will introduce 2 new packages solar and fuel2solar ( + some dependencies)
in Centos7, they will be available in mos-master repository. We would like to
include not installed solar and fuel2solar in 9.0 ISO.

  

Predicted UX:  
\- user will need to install 2 rpm packages: solar and fuel2solar  
\- user will configure everything in fuel-web and then switch to solar CLI
before clicking "deploy" button

  

Our current plan is:  
1\. create blueprint about introducing these 2 additional packages (and
requirements)  
2\. then should we describe everything again in openstack-dev ML  
3\. describe this everything in fuel docs

  

Maybe instead blueprint  in 1st step should we create full blown fuel-spec ?
That spec obviously will not require any QA activities.

  

  

\--

Warm regards,

Jędrzej Nowak

  

  

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

I definitely agree with what Evgeniy said.

@Oleg could you make a step-by-step how do you imagine this
integration (with separate ConfigDB) ? To me this adds at least one
component / integration point.


> The point is that for Solar integration, we still need integration points, 
> and the less of them we have, the more simple the transition is going to be..

With DR+DP you will have exactly one integration point.

> I also don't like polling model for data processors. In my understanding, 
> components should push their changes through the pipeline. Although this is 
> pure implementation detail and is not really important ATM.

Push is problematic because:
1) if you don't own all components then you're in troubles
2) you never know how valid are data of component (you could add some
time based markers, but still it's not enough), you basically loose
control about data. With pull model you know more about context etc.

On Mon, Dec 21, 2015 at 11:49 AM, Evgeniy L  wrote:
> Comments inline.
>
> On Mon, Dec 21, 2015 at 1:42 PM, Oleg Gelbukh  wrote:
>>
>> The problem with this approach is that we can't manage the interfaces
>> between components without changing the code of 2+ components (i.e. one that
>> provides the data and all that consume it).
>
>
> I'm not sure if understand you correctly, none of these approaches
> require to change the components, we will have change a single
> place, which is specific data processor.
>
>>
>>
>> I also don't like polling model for data processors. In my understanding,
>> components should push their changes through the pipeline. Although this is
>> pure implementation detail and is not really important ATM.
>
>
> We don't have any other choice, we don't control components, configuration
> components are not going to implement Fuel specific logic, so Fuel has
> to pool the data.
>
>>
>>
>> The point is that for Solar integration, we still need integration points,
>> and the less of them we have, the more simple the transition is going to
>> be..
>
>
> As described above, there will be a single integration point, data
> processor.
>>
>>
>> --
>> Best regards,
>> Oleg Gelbukh
>>
>> On Mon, Dec 21, 2015 at 11:32 AM, Evgeniy L  wrote:
>>>
>>> Hi Oleg,
>>>
>>> I understand the concern, but in case of integration specifically with
>>> Solar,
>>> I don't see any reasons to add ConfigDB, because Solar by itself is a
>>> ConfigDB.
>>> At the same time I would agree that there might be a case, when user uses
>>> Zookeeper/Puppet Master/CMDB as a data store, in this case we should
>>> store
>>> the data directly in those services, without keeping them in yet another
>>> storage.
>>>
>>> So the flow will look like this:
>>> Components ->
>>> Data get polled by data processors ->
>>> | Solar data processor puts the data into Solar in its format
>>> | Zookeeper data processor puts the data into Zookeeper in its format
>>> | Custom CMDB data processor puts the data into CMDB in its own format
>>>
>>> Thanks,
>>>
>>> On Fri, Dec 18, 2015 at 7:00 PM, Oleg Gelbukh 
>>> wrote:

 Hi,

 The idea behind configdb is that it is independent component that
 defines data flows between other components of the system. It has no
 knowledge about those components or specifics of their data. Data formats
 are defined by components themselves via schemas/templates and can be
 changed at any time (i.e. don't require code changes).

 Important 'pro' having ConfigDB separate from Solar is that it will
 simplify transition from current Fuel architecture by breaking it into more
 definite stages and reducing the number of components Solar have to be
 integrated with.

 --
 Best regards,
 Oleg Gelbukh

 On Wed, Dec 16, 2015 at 4:38 PM, Evgeniy L  wrote:
>
> Hi Dmitry,
>
> I also don't think that we should duplicate the data in configdb,
> because in this case there will be +2 additional interfaces which
> will require to covert the data into configdb and after that from
> configdb to Solar, which seems redundant overhead.
>
> But we should be able to put the data directly to user's
> CMDB/ZooKeeper/Puppet Master/etc.
>
> Thanks,
>
> On Wed, Dec 16, 2015 at 2:03 AM, Dmitriy Shulyak
>  wrote:
>>
>> Hello folks,
>>
>> This topic is about configuration storage which will connect data
>> sources (nailgun/bareon/others) and orchestration. And right now we are
>> developing two projects that will overlap a bit.
>>
>> I understand there is not enough context to dive into this thread
>> right away, but i will appreciate if those people, who participated in
>> design, will add their opinions/clarifications on this matter.
>>
>> Main disagreements
>> ---
>> 1. configdb should be 

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

+1 for Lukasz concerns.

But if we really need operate with "solar resources database" as a kv
store, then we could implement service on top of it, It could be then
separate project, which would work as separate service. Would it fulfill
the requirements ? (we could implement it using some already existing
protocol).

On Wed, Dec 16, 2015 at 12:06 PM, Lukasz Oles  wrote:

> Hi Dima,
>
> On Wed, Dec 16, 2015 at 12:03 AM, Dmitriy Shulyak 
> wrote:
> > Hello folks,
> >
> > This topic is about configuration storage which will connect data sources
> > (nailgun/bareon/others) and orchestration. And right now we are
> developing
> > two projects that will overlap a bit.
> What 2 projects?
>
> >
> > I understand there is not enough context to dive into this thread right
> > away, but i will appreciate if those people, who participated in design,
> > will add their opinions/clarifications on this matter.
> >
> > Main disagreements
> > ---
> > 1. configdb should be passive, writing to configdb is someone else
> > responsibility
> > + simpler implementation, easier to use
> > - we will need another component that will do writing, or split this
> > responsibility somehow
> >
> > 2. can be used without other solar components
> > + clear inteface between solar components and storage layer
> > - additional work required to design/refactor communication layer between
> > modules in solar
> > - some data will be duplicated between solar orchestrator layer and
> configdb
> >
> > 3. templates for output
> > technical detail, can be added on top of solardb if required
> Who disagrees with who and about what? It's not clear for me. As far
> as I remember we agreed to not use configdb but to use solar "data
> resources". Data would be fetched using managers  in the way as you
> are doing now in f2s branch.
> What is the problem here?
>
> Regards
>
>
> > Similar functionality
> > --
> > 1. Hierachical storage
> > 2. Versioning of changes
> > 3. Possibility to overwrite config values
> > 4. Schema for inputs
> >
> > Overall it seems that we share same goals for both services,
> > the difference lies in organizational and technical implementation
> details.
> >
> > Possible solutions
> > 
> > 1. develop configdb and solar with duplicated functionality
> > - at least 2 additional components will be added to the picture,
> > one is configdb, another one will need to sync data between configdb and
> > solar
> > - in some cases data in solar and configdb will be 100% duplicated
> > - different teams will work on same functionality
> > - integration of additional component for fuel will require integration
> with
> > configdb and with solar
> > + configdb will be independent from solar orchestration/other components
> >
> > 2. make service out of solardb, allign with configdb use cases
> > + solardb will be independent from solar orchestration/other solar
> > components
> > + integration of fuel component will be easier than in 1st version
> > + clarity about components responsibility and new architecture
> > - redesign/refactoring communication between components in solar
> >
> > 3. do not use configdb/no extraction of solardb
> > - inproc communication, which can lead to coupled components (not the
> case
> > currently)
> > + faster implementation (no major changes required for integration with
> > fuel)
> > + clarity about components responsibility and new architecture
> >
> > Summary
> > -
> > For solar it makes no difference where data will come from: configdb or
> > data sources, but in overall fuel architecture it will lead to
> significant
> > complexity increase.
> > It would be the best to follow 2nd path, because in long term we don't
> want
> > tightly coupled components, but in nearest future we need to concentrate
> > on:
> > - integration with fuel
> > - implementing policy engine
> > - polishing solar components
> > This is why i am not sure that we can spend time on 2nd path right now,
> > or even before 9.0.
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Łukasz Oleś
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
--
Warm regards,
Jędrzej Nowak
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [Fuel][Solar] SolarDB/ConfigDB place in Fuel

[Jedrzej Nowak]

On Wed, Dec 16, 2015 at 12:40 PM, Bogdan Dobrelya
 wrote:
> Data resources shall fill the configdb by results of the fetched data
> shaped by data processors aka serializers. The shaping process assumes
> applying of all versioning and schema transformations knowledge required
> to convert data from external sources into expected end state to make it
> consumable by the Policy Engine.
[..]
> Anyway, a
> solar resource just cannot be used without other solar components.

It depends what would be their usage, we could make them "like KV
store" with external access with some simple existing protocol (no
transactions, no bulk operations etc). Also it should be limited only
to DRs. Then we will have it's data, and PE would be responsible for
correct connections, DP - like in f2s - still could be possible/valid
in this case.
But as I said, it all depends on use cases.


-- 
--
Warm regards,
Jędrzej Nowak

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev