[openstack-dev] [FUEL] Authentication in FUEL meeting notes

[Alexander Kislitsky]

16.07.2014 meeting results:

Participants:
L. Oles
N. Markov
V. Kramskikh
A. Kislitsky

Acceptance points of authentication in FUEL:

   1.

   Each project (nailgun, CLI, UI, OSTF) should authenticate in Keystone
   and use received token in other services requests
   2.

   Authentication can be enabled or disabled by parameters in settings for
   each project
   3.

   We should have tests coverage with and without authentication
   4.

   We should implement default strategy(policy) for API requests -
   authentication required, but for some requests should be excluded from
   authentication checking (for example adding node, checking is
   authentication enabled in nailgun API)
   5. Nailgun API should handle Keystone tokens
   6. We should have human oriented error messages for Keystone and
   authentication errors
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [FUEL] Zabbix in MOS meeting notes

[Alexander Kislitsky]

Hi!

You are right, we are talking about
https://blueprints.launchpad.net/fuel/+spec/monitoring-system

We want to estimate required time for merging HA implementation into
current code. If it possible we will try to distribute HA solution in
current release. In other case it will be delivered in next releases. In
current implementation (non HA) Zabbix server is deployed onto separate
node.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][Nailgun] Web framework

[Alexander Kislitsky]

We had used Flask in the fuel-stats. It was easy and pleasant and all
project requirements was satisfied. And I saw difficulties and workarounds
with Pecan, when Nick integrated it into Nailgun.
So +1 for Flask.


On Tue, Dec 2, 2014 at 11:00 PM, Nikolay Markov nmar...@mirantis.com
wrote:

 Michael, we already solved all issues I described, and I just don't
 want to solve them once again after moving to another framework. Also,
 I think, nothing of these wishes contradicts with good API design.

 On Tue, Dec 2, 2014 at 10:49 PM, Michael Krotscheck
 krotsch...@gmail.com wrote:
  This sounds more like you need to pay off technical debt and clean up
 your
  API.
 
  Michael
 
  On Tue Dec 02 2014 at 10:58:43 AM Nikolay Markov nmar...@mirantis.com
  wrote:
 
  Hello all,
 
  I actually tried to use Pecan and even created a couple of PoCs, but
  there due to historical reasons of how our API is organized it will
  take much more time to implement all workarounds we need to issues
  Pecan doesn't solve out of the box, like working with non-RESTful
  URLs, reverse URL lookup, returning custom body in 404 response,
  wrapping errors to JSON automatically, etc.
 
  As far as I see, each OpenStack project implements its own workarounds
  for these issues, but still it requires much less men and hours for us
  to move to Flask-Restful instead of Pecan, because all these problems
  are already solved there.
 
  BTW, I know a lot of pretty big projects using Flask (it's the second
  most popular Web framework after Django in Python Web community), they
  even have their own hall of fame:
  http://flask.pocoo.org/community/poweredby/ .
 
  On Tue, Dec 2, 2014 at 7:13 PM, Ryan Brown rybr...@redhat.com wrote:
   On 12/02/2014 09:55 AM, Igor Kalnitsky wrote:
   Hi, Sebastian,
  
   Thank you for raising this topic again.
  
   [snip]
  
   Personally, I'd like to use Flask instead of Pecan, because first one
   is more production-ready tool and I like its design. But I believe
   this should be resolved by voting.
  
   Thanks,
   Igor
  
   On Tue, Dec 2, 2014 at 4:19 PM, Sebastian Kalinowski
   skalinow...@mirantis.com wrote:
   Hi all,
  
   [snip explanation+history]
  
   Best,
   Sebastian
  
   Given that Pecan is used for other OpenStack projects and has plenty
 of
   builtin functionality (REST support, sessions, etc) I'd prefer it for
 a
   number of reasons.
  
   1) Wouldn't have to pull in plugins for standard (in Pecan) things
   2) Pecan is built for high traffic, where Flask is aimed at much
 smaller
   projects
   3) Already used by other OpenStack projects, so common patterns can be
   reused as oslo libs
  
   Of course, the Flask community seems larger (though the average flask
   project seems pretty small).
  
   I'm not sure what determines production readiness, but it seems to
 me
   like Fuel developers fall more in Pecan's target audience than in
   Flask's.
  
   My $0.02,
   Ryan
  
   --
   Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.
  
   ___
   OpenStack-dev mailing list
   OpenStack-dev@lists.openstack.org
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
  --
  Best regards,
  Nick Markov
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 



 --
 Best regards,
 Nick Markov

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] [nailgun] [UI] network_check_status fleild for environments

[Alexander Kislitsky]

Guys, definitely we shouldn't delete tasks.

+1 for warning.

On Fri, Jan 16, 2015 at 3:58 PM, Evgeniy L e...@mirantis.com wrote:

 Hi,

 1) +1 for warning

 2) I don't think that we should delete tasks, it's a history which can be
 useful,
 for example for stats feature, also it's useful for debugging, but each
 task
 should have created_at and updated_at fields, and from api you will be able
 to get the latest tasks for specific env.

 Thanks,

 On Thu, Jan 15, 2015 at 7:20 PM, Vitaly Kramskikh vkramsk...@mirantis.com
  wrote:

 Folks,

 I want to discuss possibility to add network verification status field
 for environments. There are 2 reasons for this:

 1) One of the most frequent reasons of deployment failure is wrong
 network configuration. In the current UI network verification is completely
 optional and sometimes users are even unaware that this feature exists. We
 can warn the user before the start of deployment if network check failed of
 wasn't performed.

 2) Currently network verification status is partially tracked by status
 of the last network verification task. Sometimes its results become stale,
 and the UI removes the task. There are a few cases when the UI does this,
 like changing network settings, adding a new node, etc (you can grep
 removeFinishedNetworkTasks to see all the cases). This definitely should
 be done on backend.

 What is your opinion on this?

 --
 Vitaly Kramskikh,
 Software Engineer,
 Mirantis, Inc.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Plugins repositories moved to stackforge

[Alexander Kislitsky]

Hi, guys

All our plugins repositories are moved to stackforge. Each plugin has
separate repo and now can be maintained independently from core product
cycle and without core-team attraction.

Here is a map of migration:

https://github.com/stackforge/fuel-plugins/tree/master/external_glusterfs
- https://github.com/stackforge/fuel-plugin-external-glusterfs

https://github.com/stackforge/fuel-plugins/tree/master/external_nfs -
https://github.com/stackforge/fuel-plugin-external-nfs

https://github.com/stackforge/fuel-plugins/tree/master/ha_fencing -
https://github.com/stackforge/fuel-plugin-ha-fencing

https://github.com/stackforge/fuel-plugins/tree/master/lbaas -
https://github.com/stackforge/fuel-plugin-neutron-lbaas

https://github.com/Mirantis/fuel-plugins/tree/master/cinder_netapp -
https://github.com/stackforge/fuel-plugin-cinder-netapp

https://github.com/Mirantis/fuel-plugins/tree/master/vpnaas -
https://github.com/stackforge/fuel-plugin-neutron-vpnaas


Now external_glusterfs, external_nfs, ha_fencing, lbaas can be removed from
the https://github.com/stackforge/fuel-plugins:
https://review.openstack.org/#/c/151143/

Repository https://github.com/Mirantis/fuel-plugins/ is not required and
can be removed or made private.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Plugins repositories moved to stackforge

[Alexander Kislitsky]

CI is requirement only for certified plugins.

Is CI part of Fuel CI or not depends only on the plugin. In some cases we
just can't run plugin CI on Fuel CI environment.

On Thu, Jan 29, 2015 at 1:03 PM, Aleksandra Fedorova afedor...@mirantis.com
 wrote:

 Do we have a plan to enable CI for this repositories? Tests to run on
 commits, nightly builds, integration with nightly system tests?

 Should it be part of Fuel CI or completely independent?


 On Thu, Jan 29, 2015 at 11:55 AM, Alexander Kislitsky
 akislit...@mirantis.com wrote:
  Hi, guys
 
  All our plugins repositories are moved to stackforge. Each plugin has
  separate repo and now can be maintained independently from core product
  cycle and without core-team attraction.
 
  Here is a map of migration:
 
 
 https://github.com/stackforge/fuel-plugins/tree/master/external_glusterfs
 -
  https://github.com/stackforge/fuel-plugin-external-glusterfs
 
  https://github.com/stackforge/fuel-plugins/tree/master/external_nfs -
  https://github.com/stackforge/fuel-plugin-external-nfs
 
  https://github.com/stackforge/fuel-plugins/tree/master/ha_fencing -
  https://github.com/stackforge/fuel-plugin-ha-fencing
 
  https://github.com/stackforge/fuel-plugins/tree/master/lbaas -
  https://github.com/stackforge/fuel-plugin-neutron-lbaas
 
  https://github.com/Mirantis/fuel-plugins/tree/master/cinder_netapp -
  https://github.com/stackforge/fuel-plugin-cinder-netapp
 
  https://github.com/Mirantis/fuel-plugins/tree/master/vpnaas -
  https://github.com/stackforge/fuel-plugin-neutron-vpnaas
 
 
  Now external_glusterfs, external_nfs, ha_fencing, lbaas can be removed
 from
  the https://github.com/stackforge/fuel-plugins:
  https://review.openstack.org/#/c/151143/
 
  Repository https://github.com/Mirantis/fuel-plugins/ is not required
 and can
  be removed or made private.
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 



 --
 Aleksandra Fedorova
 Fuel Devops Engineer
 bookwar

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Additional user account in the OpenStack for fetching OpenStack workloads

[Alexander Kislitsky]

Folks,

We are collecting OpenStack workloads stats. For authentication in the
keystone we are using admin user credentials from Nailgun. Credentials can
be changed directly in the OpenStack and we will loose possibility of
fetching information.

This issue can be fixed by creation additional user account:

   1. I propose to generate additional user credentials after master node
   is installed and store it into master_node_settings table in the Nailgun.
   2. Add abstraction layer into
   
https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/statistics/utils.py#L47
   for creating additional user in the OpenStack if it isn't exists.

But this additional user can be useful for other purposes and may be we
should save credentials in other place (settings.yaml for example). And may
be creation of the additional user should be implemented outside of stats
collecting feature and may be outside of Nailgun.

Please share your thoughts on this.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] development tools

[Alexander Kislitsky]

+1 for moving fuel_development into separate repo.

On Thu, Mar 19, 2015 at 5:02 PM, Evgeniy L e...@mirantis.com wrote:

 Hi folks,

 I agree, lets create separate repo with its own cores and remove
 fuel_development from fuel-web.

 But in this case I'm not sure if we should merge the patch which
 has links to non-stackforge repositories, because location is going
 to be changed soon.

 Also it will be cool to publish it on pypi.

 Thanks,

 On Thu, Mar 19, 2015 at 4:21 PM, Sebastian Kalinowski 
 skalinow...@mirantis.com wrote:

 As I wrote in the review already: I like the idea of merging
 those two tools and making a separate repository. After that
 we could make they more visible in our documentation and wiki
 so they could benefit from being used by broader audience.

 Same for vagrant configuration - if it's useful (and it is
 since newcomers are using them) we could at least move under
 Mirantis organization on Github.

 Best,
 Seabastian


 2015-03-19 13:49 GMT+01:00 Przemyslaw Kaminski pkamin...@mirantis.com:

 Hello,

 Some time ago I wrote some small tools that make Fuel development easier
 and it was suggested to add info about them to the documentation --
 here's the review link [1].

 Evgenyi Li correctly pointed out that we already have something like
 fuel_development already in fuel-web. I think though that we shouldn't
 mix such stuff directly into fuel-web, I mean we recently migrated CLI
 to a separate repo to make fuel-web thinner.

 So a suggestion -- maybe make these tools more official and create
 stackforge repos for them? I think dev ecosystem could benefit by having
 some standard way of dealing with the ISO (for example we get questions
 from people how to apply new openstack.yaml config to the DB).

 At the same time we could get rid of fuel_development and merge that
 into the new repos (it has the useful 'revert' functionality that I
 didn't think of :))

 P.

 [1] https://review.openstack.org/#/c/140355/9/docs/develop/env.rst


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Transaction scheme

[Alexander Kislitsky]

Hi!

The refactoring of transactions management in Nailgun is critically
required for scaling.

First of all I propose to wrap HTTP handlers by begin/commit/rollback
decorator.
After that we should introduce transactions wrapping decorator into Task
execute/message calls.
And the last one is the wrapping of receiver calls.

As result we should have begin/commit/rollback calls only in transactions
decorator.

Also I propose to separate working with DB objects into separate lair and
use only high level Nailgun objects in the code and tests. This work was
started long time ago, but not finished yet.

On Thu, Apr 30, 2015 at 12:36 PM, Roman Prykhodchenko m...@romcheg.me wrote:

 Hi folks!

 Recently I faced a pretty sad fact that in Nailgun there’s no common
 approach to manage transactions. There are commits and flushes in random
 places of the code and it used to work somehow just because it was all
 synchronous.

 However, after just a few of the subcomponents have been moved to
 different processes, it all started producing races and deadlocks which are
 really hard to resolve because there is absolutely no way to predict how a
 specific transaction is managed but by analyzing the source code. That is
 rather an ineffective and error-prone approach that has to be fixed before
 it became uncontrollable.

 Let’s arrange a discussions to design a document which will describe where
 and how transactions are managed and refactor Nailgun according to it in
 7.0. Otherwise results may be sad.


 - romcheg


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Transaction scheme

[Alexander Kislitsky]

I mean, that we should have explicitly wrapped http handlers. For example:

@transaction
def PUT(...):
  ...

We don't need transactions, for example, in GET methods.

I propose to rid of complex data flows in our code. Code with 'commit' call
inside the the method should be split into independent units.

I like the solution with sending tasks to Astute at the end of handler
execution.

On Wed, May 6, 2015 at 12:57 PM, Igor Kalnitsky ikalnit...@mirantis.com
wrote:

  First of all I propose to wrap HTTP handlers by begin/commit/rollback

 I don't know what you are talking about, but we do wrap handlers in
 transaction for a long time. Here's the code

 https://github.com/stackforge/fuel-web/blob/2de3806128f398d192d7e31f4ca3af571afeb0b2/nailgun/nailgun/api/v1/handlers/base.py#L53-L84

 The issue is that we sometimes perform `.commit()` inside the code
 (e.g. `task.execute()`) and therefore it's hard to predict which data
 are committed and which are not.

 In order to avoid this, we have to declare strict scopes for different
 layers. Yes, we definitely should base on idea that handlers should
 open transaction on the begin and close it on the end. But that won't
 solve all problems, because sometimes we should commit data before
 handler's end. For instance, commit some task before sending message
 to Astute. Such cases complicate the things.. and it would be cool if
 could avoid them by re-factoring our architecture. Perhaps, we could
 send tasks to Astute when the handler is done? What do you think?

 Thanks,
 igor

 On Wed, May 6, 2015 at 12:15 PM, Lukasz Oles lo...@mirantis.com wrote:
  On Wed, May 6, 2015 at 10:51 AM, Alexander Kislitsky
  akislit...@mirantis.com wrote:
  Hi!
 
  The refactoring of transactions management in Nailgun is critically
 required
  for scaling.
 
  First of all I propose to wrap HTTP handlers by begin/commit/rollback
  decorator.
  After that we should introduce transactions wrapping decorator into Task
  execute/message calls.
  And the last one is the wrapping of receiver calls.
 
  As result we should have begin/commit/rollback calls only in
 transactions
  decorator.
 
  Big +1 for this. I always wondered why we don't have it.
 
 
 
  Also I propose to separate working with DB objects into separate lair
 and
  use only high level Nailgun objects in the code and tests. This work was
  started long time ago, but not finished yet.
 
  On Thu, Apr 30, 2015 at 12:36 PM, Roman Prykhodchenko m...@romcheg.me
 wrote:
 
  Hi folks!
 
  Recently I faced a pretty sad fact that in Nailgun there’s no common
  approach to manage transactions. There are commits and flushes in
 random
  places of the code and it used to work somehow just because it was all
  synchronous.
 
  However, after just a few of the subcomponents have been moved to
  different processes, it all started producing races and deadlocks
 which are
  really hard to resolve because there is absolutely no way to predict
 how a
  specific transaction is managed but by analyzing the source code. That
 is
  rather an ineffective and error-prone approach that has to be fixed
 before
  it became uncontrollable.
 
  Let’s arrange a discussions to design a document which will describe
 where
  and how transactions are managed and refactor Nailgun according to it
 in
  7.0. Otherwise results may be sad.
 
 
  - romcheg
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
  --
  Łukasz Oleś
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Transaction scheme

[Alexander Kislitsky]

Let's collect work items together:

   1. Introduce UnitOfWork for generated astute messages. All messages
   should be sent to Astute only after HTTP handler is finished. For example
   in ApplyChangesTaskManager we are casting messages during processing method
   '_execute_async_content'
   2. All nailgun Task objects should implement single interface. Now we
   have 'execute', 'message', 'get_message_body' methods. Task should
   implement 'message' and not cast messages to Astute. Tasks main function is
   to serialize messages for Astute. Nothing more.
   3. We should improve deadlock detecting in the Nailgun. We should trace
   not only order of tables in the locks chain but also ids of locked objects.
   Also deadlock detecting should consider update, delete queries.
   4. Commit calls should be present only in the HTTP middleware. We
   shouldn't manipulate DB transactions inside business logic.


On Wed, May 6, 2015 at 6:32 PM, Igor Kalnitsky ikalnit...@mirantis.com
wrote:

  We don't need transactions, for example, in GET methods.

 It doesn't matter whether want we or not. The SQLAlchemy implicitly
 starts transaction on first select query and it's ok. I mean,
 perhaps it's not ok, but definitely it won't lead to great performance
 degradation. A large number of projects prove this.

  I propose to rid of complex data flows in our code. Code with 'commit'
  call inside the the method should be split into independent units.

 Agree. We should get rid of non-obvious and unexpected commits.

  I like the solution with sending tasks to Astute at the end of handler
  execution.

 I don't know how much effort we should apply to implement this, but on
 first look it seems ok. I mean, we save tasks to send in some queue
 and then send them if and only iff HTTP handler reports success.
 Currently, we send it, but there are few places where HTTP handler may
 fail, report error and perform partial rollback (why partial? because
 we commit task before sending to Astute) and it looks weird. :(



 On Wed, May 6, 2015 at 1:22 PM, Alexander Kislitsky
 akislit...@mirantis.com wrote:
  I mean, that we should have explicitly wrapped http handlers. For
 example:
 
  @transaction
  def PUT(...):
...
 
  We don't need transactions, for example, in GET methods.
 
  I propose to rid of complex data flows in our code. Code with 'commit'
 call
  inside the the method should be split into independent units.
 
  I like the solution with sending tasks to Astute at the end of handler
  execution.
 
  On Wed, May 6, 2015 at 12:57 PM, Igor Kalnitsky ikalnit...@mirantis.com
 
  wrote:
 
   First of all I propose to wrap HTTP handlers by begin/commit/rollback
 
  I don't know what you are talking about, but we do wrap handlers in
  transaction for a long time. Here's the code
 
 
 https://github.com/stackforge/fuel-web/blob/2de3806128f398d192d7e31f4ca3af571afeb0b2/nailgun/nailgun/api/v1/handlers/base.py#L53-L84
 
  The issue is that we sometimes perform `.commit()` inside the code
  (e.g. `task.execute()`) and therefore it's hard to predict which data
  are committed and which are not.
 
  In order to avoid this, we have to declare strict scopes for different
  layers. Yes, we definitely should base on idea that handlers should
  open transaction on the begin and close it on the end. But that won't
  solve all problems, because sometimes we should commit data before
  handler's end. For instance, commit some task before sending message
  to Astute. Such cases complicate the things.. and it would be cool if
  could avoid them by re-factoring our architecture. Perhaps, we could
  send tasks to Astute when the handler is done? What do you think?
 
  Thanks,
  igor
 
  On Wed, May 6, 2015 at 12:15 PM, Lukasz Oles lo...@mirantis.com
 wrote:
   On Wed, May 6, 2015 at 10:51 AM, Alexander Kislitsky
   akislit...@mirantis.com wrote:
   Hi!
  
   The refactoring of transactions management in Nailgun is critically
   required
   for scaling.
  
   First of all I propose to wrap HTTP handlers by begin/commit/rollback
   decorator.
   After that we should introduce transactions wrapping decorator into
   Task
   execute/message calls.
   And the last one is the wrapping of receiver calls.
  
   As result we should have begin/commit/rollback calls only in
   transactions
   decorator.
  
   Big +1 for this. I always wondered why we don't have it.
  
  
  
   Also I propose to separate working with DB objects into separate lair
   and
   use only high level Nailgun objects in the code and tests. This work
   was
   started long time ago, but not finished yet.
  
   On Thu, Apr 30, 2015 at 12:36 PM, Roman Prykhodchenko m...@romcheg.me
 
   wrote:
  
   Hi folks!
  
   Recently I faced a pretty sad fact that in Nailgun there’s no common
   approach to manage transactions. There are commits and flushes in
   random
   places of the code and it used to work somehow just because it was
 all
   synchronous.
  
   However, after just a few

Re: [openstack-dev] [Fuel] Nominate Andrey Skedzinskiy for fuel-qa(devops) core

[Alexander Kislitsky]

Andrew shows great attention to the details. +1 for him.

On Mon, Apr 13, 2015 at 11:22 AM, Anastasia Urlapova aurlap...@mirantis.com
 wrote:

 Guys,
 I would like to nominate Andrey Skedzinskiy[1] for
 fuel-qa[2]/fuel-devops[3] core team.

 Andrey is one of the strongest reviewers, under his watchful eye are such
 features as:
 - updrade/rollback master node
 - collect usage information
 - OS patching
 - UI tests
 and others

 Please vote for Andrey!


 Nastya.

 [1]http://stackalytics.com/?project_type=stackforgeuser_id=asledzinskiy
 [2]https://github.com/stackforge/fuel-qa
 [3]https://github.com/stackforge/fuel-devops

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Packaged Fuel and Feature Groups

[Alexander Kislitsky]

Also feature_groups is used for selecting usage statistics collector. If
'mirantis' is in feature_groups we are using one instance of statistics
collector and another collector instance in other case.

On Thu, Jun 18, 2015 at 7:04 PM, Vitaly Kramskikh vkramsk...@mirantis.com
wrote:

 Hi,

 Yes, it is possible to change the value of feature_groups after master
 node installation. Currently it affects only availability of a few options
 in Fuel UI.

 2015-06-18 17:11 GMT+03:00 Aleksandra Fedorova afedor...@mirantis.com:

 Hi, everyone,

 could you please clarify a bit about how FEATURE_GROUPS [1] parameter
 is handled in Fuel.

 Currently we have it specified at ISO build stage, while from the
 description of it it looks like it is just a UI switch which doesn't
 change anything deep inside the code.

 Can it be changed at master node deployment stage, after master node
 deployment?

 Can we use exactly the same nailgun package to install all the
 different flavors just by changing configuration parameter after
 package installation?

 [1]
 https://ci.fuel-infra.org/job/merged-fuel-specs/Fuel_Development_Specs_build_results/specs/5.1/feature-groups.html#feature-groups

 --
 Aleksandra Fedorova
 Fuel Devops Engineer
 bookwar

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Vitaly Kramskikh,
 Fuel UI Tech Lead,
 Mirantis, Inc.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Getting rid of launchpad group fuel-astute

[Alexander Kislitsky]

Dear colleagues,

I'd like to get rid of group fuel-astute on launchpad. In group only 2
active members. Actually they are members of fuel-python team. Bugs for
fuel-astute project always concern to fuel-web project. Bugs assigned to
fuel-astute can stay without attention for a long time. Thus I propose to
use fuel-python team instead fuel-astute.

First of all we should reassign team for bugs [1]. After that we can remove
or disable fuel-astute launchpad group.

What do you think about this?

[1] https://goo.gl/ap35t9
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Nominate Artem Roma for fuel-ostf core

[Alexander Kislitsky]

+1

On Wed, Dec 23, 2015 at 8:15 PM, Aleksey Kasatkin 
wrote:

> +1
>
> Aleksey Kasatkin
>
>
> On Wed, Dec 23, 2015 at 5:28 PM, Aleksandr Didenko 
> wrote:
>
>> +1
>>
>> On Wed, Dec 23, 2015 at 4:21 PM, Andrey Sledzinskiy <
>> asledzins...@mirantis.com> wrote:
>>
>>> +1
>>>
>>> On Wed, Dec 23, 2015 at 5:05 PM, Tatyana Leontovich <
>>> tleontov...@mirantis.com> wrote:
>>>
 Hi guys,
 I'd like to nominate Artem Roma [0] for fuel-ostf core.

 Artem  cares about fuel-ostf more then 2 years, always provide a great
 reviews(always thorough and consistent and comes in time), extend unit
 tests coverages and provide a lot of bug fixes and improvements there.

 Please, vote for Artem!


 http://stackalytics.com/?user_id=aroma-x=all_type=all=fuel-ostf


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>>
>>>
>>> --
>>> Thanks,
>>> Andrey Sledzinskiy
>>> QA Engineer,
>>> Mirantis, Kharkiv
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] [Nailgun] Deadlocks and random test failures

[Alexander Kislitsky]

Hi, guys.

Igor is absolutelly right - there are no deadlocks. We have only warnings
from detector, but they caused by difference between actual locking order
in the code and allowed by detector. It is annoying, but detection is used
only in development environment, thus it is not high bug.

When DB deadlock occurs we have ShareLock exception in the logs and it
raised before detector warning. So we always have deadlock exception in the
logs if it occurs.
I think tests failures are caused by another issue. As I can see we have a
set of random failures in the tests now and bugs on it:

https://bugs.launchpad.net/fuel/+bug/1437232
https://bugs.launchpad.net/fuel/+bug/1502908
https://bugs.launchpad.net/fuel/+bug/1518268
https://bugs.launchpad.net/fuel/+bug/1521966

We should focus on fixing these bugs. Could you please help us with
detection of the root cause of UI tests failures? May be we have another
floating bug in tests.

On Wed, Dec 30, 2015 at 4:11 PM, Igor Kalnitsky <ikalnit...@mirantis.com>
wrote:

> Hey Vitaly,
>
> Are you the problem is in deadlock? I see the deadlock detecter
> traceback, but not an actual deadlock.
>
> I'm not sure could it be a reason for failure or not, it's better to
> ask Alexander Kislitsky.
>
> Thanks,
> Igor
>
> On Wed, Dec 30, 2015 at 2:57 PM, Vitaly Kramskikh
> <vkramsk...@mirantis.com> wrote:
> > Hi,
> >
> > We have a long-living issue with deadlocks in nailgun which used to
> almost
> > harmless and caused rare test failures. But test failures become more
> > frequent and today there is ~20% probability that they will fail on a
> > working code, which is really annoying. Moreover, a few weeks ago it
> started
> > to affect UI functional tests: cluster reset task may hang, so this issue
> > now may affect real deployments.
> >
> > I think we need to do something with it ASAP.
> >
> > --
> > Vitaly Kramskikh,
> > Fuel UI Tech Lead,
> > Mirantis, Inc.
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Task Based Deployment Is at Least Twice Faster

[Alexander Kislitsky]

+1

On Tue, Feb 9, 2016 at 12:28 PM, Anastasia Urlapova 
wrote:

> +1
>
> On Tue, Feb 9, 2016 at 11:51 AM, Evgeniy L  wrote:
>
>> +1
>>
>> On Mon, Feb 8, 2016 at 7:58 PM, Vladimir Kozhukalov <
>> vkozhuka...@mirantis.com> wrote:
>>
>>> +1 to enable it ASAP.
>>>
>>> It will also affect our deployment tests (~1 hour vs. ~2.5 hours).
>>>
>>> Vladimir Kozhukalov
>>>
>>> On Mon, Feb 8, 2016 at 7:35 PM, Bulat Gaifullin >> > wrote:
>>>
 +1.

 Regards,
 Bulat Gaifullin
 Mirantis Inc.



 > On 08 Feb 2016, at 19:05, Igor Kalnitsky 
 wrote:
 >
 > Hey Fuelers,
 >
 > When we are going to enable it? I think since HCF is passed for
 > stable/8.0, it's time to enable task-based deployment for master
 > branch.
 >
 > Opinion?
 >
 > - Igor
 >
 > On Wed, Feb 3, 2016 at 12:31 PM, Bogdan Dobrelya <
 bdobre...@mirantis.com> wrote:
 >> On 02.02.2016 17:35, Alexey Shtokolov wrote:
 >>> Hi Fuelers!
 >>>
 >>> As you may be aware, since [0] Fuel has implemented a new
 orchestration
 >>> engine [1]
 >>> We switched the deployment paradigm from role-based (aka granular)
 to
 >>> task-based and now Fuel can deploy all nodes simultaneously using
 >>> cross-node dependencies between deployment tasks.
 >>
 >> That is great news! Please do not forget about docs updates as well.
 >> Those docs are always forgotten like poor orphans... I submitted a
 patch
 >> [0] to MOS docs, please review and add more details, if possible, for
 >> plugins impact as well.
 >>
 >> [0] https://review.fuel-infra.org/#/c/16509/
 >>
 >>>
 >>> This feature is experimental in Fuel 8.0 and will be enabled by
 default
 >>> for Fuel 9.0
 >>>
 >>> Allow me to show you the results. We made some benchmarks on our
 bare
 >>> metal lab [2]
 >>>
 >>> Case #1. 3 controllers + 7 computes w/ ceph.
 >>> Task-based deployment takes *~38* minutes vs *~1h15m* for granular
 (*~2*
 >>> times faster)
 >>> Here and below the deployment time is average time for 10 runs
 >>>
 >>> Case #2. 3 controllers + 3 mongodb + 4 computes w/ ceph.
 >>> Task-based deployment takes *~41* minutes vs *~1h32m* for granular
 >>> (*~2.24* times faster)
 >>>
 >>>
 >>>
 >>> Also we took measurements for Fuel CI test cases. Standard BVT
 (Master
 >>> node + 3 controllers + 3 computes w/ ceph. All are in qemu VMs on
 one host)
 >>>
 >>> Fuel CI slaves with *4 *cores *~1.1* times faster
 >>> In case of 4 cores for 7 VMs they are fighting for CPU resources
 and it
 >>> marginalizes the gain of task-based deployment
 >>>
 >>> Fuel CI slaves with *6* cores *~1.6* times faster
 >>>
 >>> Fuel CI slaves with *12* cores *~1.7* times faster
 >>
 >> These are really outstanding results!
 >> (tl;dr)
 >> I believe the next step may be to leverage the "external install &
 svc
 >> management" feature (example [1]) of the Liberty release (7.0.0) of
 >> Puppet-Openstack (PO) modules. So we could use separate concurrent
 >> cross-depends based tasks *within a single node* as well, like:
 >> - task: install_all_packages - a singleton task for a node,
 >> - task: [configure_x, for each x] - concurrent for a node,
 >> - task: [manage_service_x, for each x] - some may be concurrent for a
 >> node, while another shall be serialized.
 >>
 >> So, one might use the "--tags" separator for concurrent puppet runs
 to
 >> make things go even faster, for example:
 >>
 >> # cat test.pp
 >> notify
 >> {"A": tag => "a" }
 >> notify
 >> {"B": tag => "b" }
 >>
 >> # puppet apply test.pp
 >> Notice: A
 >> Notice: /Stage[main]/Main/Notify[A]/message: defined 'message' as 'A'
 >> Notice: B
 >> Notice: /Stage[main]/Main/Notify[B]/message: defined 'message' as 'B'
 >>
 >> # puppet apply test.pp --tags a
 >> Notice: A
 >> Notice: /Stage[main]/Main/Notify[A]/message: defined 'message' as 'A'
 >>
 >> # puppet apply test.pp --tags a & puppet apply test.pp --tags b
 >> Notice: B
 >> Notice: /Stage[main]/Main/Notify[B]/message: defined 'message' as 'B'
 >> Notice: A
 >> Notice: /Stage[main]/Main/Notify[A]/message: defined 'message' as 'A'
 >>
 >> Which is supposed to be faster, although not for this example.
 >>
 >> [1] https://review.openstack.org/#/c/216926/3/manifests/init.pp
 >>
 >>>
 >>> You can see additional information and charts in the presentation
 [3].
 >>>
 >>> [0]
 >>> -
 http://lists.openstack.org/pipermail/openstack-dev/2015-December/082093.html
 >>> [1]
 >>> -
 

Re: [openstack-dev] [Fuel][Fuel-Library] Nominating Matthew Mosesohn for Fuel Library Core

[Alexander Kislitsky]

+1

On Thu, Feb 25, 2016 at 12:32 PM, Bulat Gaifullin 
wrote:

> +1
>
> Regards,
> Bulat Gaifullin
> Mirantis Inc.
>
>
>
> On 24 Feb 2016, at 16:02, Aleksandr Didenko  wrote:
>
> +1
>
> On Wed, Feb 24, 2016 at 1:50 PM, Vladimir Kuklin 
> wrote:
>
>> Fellow Fuelers
>>
>> I would like to kindly ask you to consider voting for Matthew Mosesohn as
>> a Fuel Library Core
>> reviewer.
>>
>> Matthew has been working with Fuel since its inception, worked on
>> countless amount of features, such as :
>>
>> Master Node Upgrades and Backup
>> Improvements to Fuel Infra
>> Mitaka, Kilo and Juno Support
>> Detachable Services Plugins
>> RHOS Support in Fuel
>> UCA Support
>> Refactoring of Haproxy and Firewall pieces
>>
>> He has also been known as our Fuel Master node and Docker guru and has
>> been continuously working on bugfixing where he scores as a bug squashing
>> monster with more than 150 bugfixes to all the parts of Fuel Library (#1
>> throughout FL history).
>>
>> As you can see, there is not a piece of Fuel Library that he has not yet
>> gained experience with.
>>
>> And this can easily be fetched with simple statistics of Matt's
>> contribution. He is the topmost contributor to all Fuel projects among all
>> Fuel Library folks and is the 3rd contributor of Fuel Library.
>> He also reviews a lot and has a fair amount of -1's (he is not as cruel
>> as me, though :-))
>>
>> Having that said, I would like to open the vote to promote Matt to
>> OpenStack Fuel Library core reviewers.
>>
>> --
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Fuel Library Tech Lead,
>> Mirantis, Inc.
>> +7 (495) 640-49-04
>> +7 (926) 702-39-68
>> Skype kuklinvv
>> 35bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com 
>> www.mirantis.ru
>> vkuk...@mirantis.com
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Nominating Alexander Kislitsky for fuel-web-core

[Alexander Kislitsky]

Thank you!

On Mon, Feb 27, 2017 at 1:29 PM, Ihor Kalnytskyi <i...@kalnytskyi.com>
wrote:

> I'm pleased to welcome Alexander to the fuel-web core team!
>
> https://review.openstack.org/#/admin/groups/664,members
>
> On Wed, Feb 22, 2017 at 10:40 AM, Fedor Zhadaev <fzhad...@mirantis.com>
> wrote:
> > +1
> >
> > On Wed, Feb 22, 2017 at 12:36 AM Булат Гайфуллин <gaifulli...@gmail.com>
> > wrote:
> >>
> >> +1
> >>
> >> 2017-02-21 17:01 GMT+03:00 Alexey Shtokolov <ashtoko...@mirantis.com>:
> >>>
> >>> Hey fellow fuelers,
> >>>
> >>> I'd like to nominate Alexander Kislitsky to the fuel-web-core team.
> >>> He's doing thorough review [1], participate in feature developments
> >>> (e.g. Config-DB enhancements, network-manager performance
> >>> improvements) and made an outstanding contribution bug-fixing.
> >>>
> >>> Core reviewers, please reply back with +1/-1.
> >>>
> >>> Thanks,
> >>> Ihor
> >>>
> >>> [1] http://stackalytics.com/?release=ocata=fuel-web
> >>>
> >>>
> >>> 
> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >> 
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > --
> > Fedor Zhadaev
> > email: fzhad...@mirantis.com
> > skype: zhadaevfm
> > IRC: fzhadaev
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev