Re: [openstack-dev] [Fuel] snapshot tool

[Dmitry Sutyagin]

Team,

A "bicycle" will have to be present anyway, as a code which interacts with
Ansible, because as far as I understand Ansible on it's own cannot provide
all the functionality in one go, so a wrapper for it will have to be
present anyway.

I think me and Alexander we will look into converting Timmy into
Ansible-based tool. One way to go would be to make Ansible a backend option
for Timmy (ssh being the alternative).

I agree that the folder-driven structure is not easy to manipulate, but you
don't want to put all your scripts inside Ansible playbooks, that would
also be a mess. Something in-between would work well - folder structure for
available
scripts, and playbooks which link to them via -script: ,
generated statically (default) or dynamically if need be.

Also, I imagine some functions might not be directly possible with Ansible,
such as parallel stdout delivery of binary data into separate files (Timmy
pulls logs compressed on the fly on the node side through ssh, to avoid
using any unnecessary disk space on env nodes and local machine). So again,
for maximum efficiency and specifc tasks a separate tool might be required,
apart of Ansible.



On Wed, Apr 20, 2016 at 5:36 PM, Dmitriy Novakovskiy <
dnovakovs...@mirantis.com> wrote:

> There's a thread on openstack-dev, but
> - nobody replied there (I checked this morning)
> - I can't link PROD tickets there :)
>
>
> On Thursday, April 21, 2016, Mike Scherbakov 
> wrote:
>
>> Guys,
>> how did it turn into openstack-dev from mos-dev, without any tags and
>> original messages... ?
>>
>> Please be careful when replying... There is a different email thread
>> started in OpenStack dev, with [Fuel] in subject..
>>
>> On Wed, Apr 20, 2016 at 10:08 AM Dmitry Nikishov 
>> wrote:
>>
>>> Dmitry,
>>>
>>> I mean, currently shotgun fetches services' configuration along with
>>> astute.yaml. These files contain passwords, keys, tokens. I beleive, these
>>> should be sanitized. Or, better yet, there should be an option to sanitize
>>> sensitive data from fetched files.
>>>
>>>
>>> Aleksandr,
>>>
>>> Currently Fuel has a service non-root account with passwordless sudo
>>> enabled. This may change in the future (the passwordless part), however,
>>> now I don't see an issue there.
>>> Additionally, it is possible for users to configure sudo for the
>>> user-facing account however they like.
>>>
>>> In regards to have this tool to use a non-root accounts, there are 2
>>> items:
>>> - execute commands, that require elevated privileges (the easy part --
>>> user has to be able to execute these commands with sudo and without
>>> password)
>>> - copy files, that this user doesn't have read privileges for.
>>>
>>> For the second item, there are 2 possible solutions:
>>> 1. Give the non-root user read privileges for these files.
>>> Pros:
>>> - More straightforward, generally acceptable way
>>> Cons:
>>> - Requires additional implementation to give permissions to the user
>>> - (?) Not very extensible: to allow copying a new file, we'd have to
>>> first add it to the tool's config, and somehow implement adding read
>>> permissions
>>>
>>> 2. Somehow allow to copy these files with sudo.
>>> Pros:
>>> - More simple implementation: we'll just need to make sure that the user
>>> can do passwordless sudo
>>> - Extensible: to add more files, it's enough to just specify them in the
>>> tool's configuration.
>>> Cons:
>>> - Non-obvious, obscure way
>>> - Relies on having to be able to do something like "sudo cat
>>> /path/to/file", which is not much better that just giving the user read
>>> privileges. In fact, the only difference between this and giving the user
>>> the read rights is that it is possible to allow "sudo cat" for files, that
>>> don't yet exist, whereas giving permissions requires that these files
>>> already are on the filesystem.
>>>
>>> What way do you think is more appropriate?
>>>
>>>
>>> On Wed, Apr 20, 2016 at 5:28 AM, Aleksandr Dobdin 
>>> wrote:
>>>
 Dmitry,

 You can create a non-root user account without root privileges but you
 need to add it to appropriate groups and configure sudo permissions (even
 though you add this user to root group, it will fail with iptables command
 for example) to get config files and launch requested commands.I
 suppose that it is possible to note this possibility in the documentation
 and provide a customer with detailed instructions on how to setup this user
 account.There are some logs that will also be missing from the
 snapshot with the message permission denied (only the root user has
 access to some files with 0600 mask)
 This user account could be specified into config.yaml (ssh -> opts
 option)

 Sincerely yours,
 Aleksandr Dobdin
 Senior Operations Engineer
 Mirantis
 ​Inc.​



 __

Re: [openstack-dev] [Fuel] snapshot tool

[Dmitry Nikishov]

Dmitry,

I mean, currently shotgun fetches services' configuration along with
astute.yaml. These files contain passwords, keys, tokens. I beleive, these
should be sanitized. Or, better yet, there should be an option to sanitize
sensitive data from fetched files.


Aleksandr,

Currently Fuel has a service non-root account with passwordless sudo
enabled. This may change in the future (the passwordless part), however,
now I don't see an issue there.
Additionally, it is possible for users to configure sudo for the
user-facing account however they like.

In regards to have this tool to use a non-root accounts, there are 2 items:
- execute commands, that require elevated privileges (the easy part -- user
has to be able to execute these commands with sudo and without password)
- copy files, that this user doesn't have read privileges for.

For the second item, there are 2 possible solutions:
1. Give the non-root user read privileges for these files.
Pros:
- More straightforward, generally acceptable way
Cons:
- Requires additional implementation to give permissions to the user
- (?) Not very extensible: to allow copying a new file, we'd have to first
add it to the tool's config, and somehow implement adding read permissions

2. Somehow allow to copy these files with sudo.
Pros:
- More simple implementation: we'll just need to make sure that the user
can do passwordless sudo
- Extensible: to add more files, it's enough to just specify them in the
tool's configuration.
Cons:
- Non-obvious, obscure way
- Relies on having to be able to do something like "sudo cat
/path/to/file", which is not much better that just giving the user read
privileges. In fact, the only difference between this and giving the user
the read rights is that it is possible to allow "sudo cat" for files, that
don't yet exist, whereas giving permissions requires that these files
already are on the filesystem.

What way do you think is more appropriate?


On Wed, Apr 20, 2016 at 5:28 AM, Aleksandr Dobdin 
wrote:

> Dmitry,
>
> You can create a non-root user account without root privileges but you
> need to add it to appropriate groups and configure sudo permissions (even
> though you add this user to root group, it will fail with iptables command
> for example) to get config files and launch requested commands.I suppose
> that it is possible to note this possibility in the documentation and
> provide a customer with detailed instructions on how to setup this user
> account.There are some logs that will also be missing from the snapshot
> with the message permission denied (only the root user has access to some
> files with 0600 mask)
> This user account could be specified into config.yaml (ssh -> opts option)
>
> Sincerely yours,
> Aleksandr Dobdin
> Senior Operations Engineer
> Mirantis
> ​Inc.​
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Dmitry Nikishov,
Deployment Engineer,
Mirantis, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] snapshot tool

[Aleksandr Dobdin]

Dmitry,

You can create a non-root user account without root privileges but you need
to add it to appropriate groups and configure sudo permissions (even though
you add this user to root group, it will fail with iptables command for
example) to get config files and launch requested commands.I suppose that
it is possible to note this possibility in the documentation and provide a
customer with detailed instructions on how to setup this user account.There
are some logs that will also be missing from the snapshot with the
message permission
denied (only the root user has access to some files with 0600 mask)
This user account could be specified into config.yaml (ssh -> opts option)

Sincerely yours,
Aleksandr Dobdin
Senior Operations Engineer
Mirantis
​Inc.​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] snapshot tool

[Dmitry Sutyagin]

IMHO, removal of sensitive information is done by services when they (do
not) log relative data to logs, such as tokens. Current set of commands
only collects specific config folders and files and logs, but if an admin
decided to store keys in one of these folders - the tool will collect them
too. It's up to the end user to only provide data collected via this tool
to a trusted party. Same goes for our current snapshot mechanism.
As of sanitization for hostnames, IPs, etc - this will make the diagnostic
snapshot pretty useless because it's important for navigation within logs
and configs, for RCA compilation, etc.

I cannot say much about running under a non-root account, I guess that
would be pretty easy to implement, let's wait for Alexander's reply. I am
not sure it is useful though because a non-root user will not have
necessary access unless there is a passwordless non-interactive sudo config.

On Tue, Apr 19, 2016 at 1:39 PM, Dmitry Nikishov 
wrote:

> Hello,
>
> I've got a couple of questions:
> - What about this tool using non-root accounts to connect to OpenStack
> nodes? Currently, it seems to assume that it always is going to use "root"
> for SSH.
> - Shouldn't it sanitize all sensitive information (user names, host names,
> passwords, tokens, keys etc)?
>
> Thanks.
>
> On Tue, Apr 19, 2016 at 4:52 AM, Aleksandr Dobdin 
> wrote:
>
>> Hello team,
>>
>> I want to discuss the tool  that we
>> have created for MOS as a replacement/alternative of shotgun.
>>
>>
>>
>>-
>>
>>The tool is based on
>>https://etherpad.openstack.org/p/openstack-diagnostics
>>-
>>
>>Should work fine on the following environments that were tested: 4.x,
>>5.x, 6.x, 7.0, 8.0
>>-
>>
>>Operates non-destructively.
>>-
>>
>>Can be launched on any host within admin network, provided the fuel
>>node IP is specified and access is possible to Fuel and other nodes via 
>> ssh
>>from local system.
>>-
>>
>>Parallel launch, only on the nodes that are 'online'. Some filters
>>for nodes are also available.
>>-
>>
>>Commands (from ./cmds directory) are separated according to roles
>>(detected automatically) by the symlinks. Thus, the command list may 
>> depend
>>on release, roles and OS. In addition, there can be some commands that run
>>everywhere. There are also commands that are executed only on one node
>>according to its role, using the first node of this type they
>>encounter.
>>-
>>
>>Modular: possible to create a special package that contains only
>>certain required commands.
>>-
>>
>>Collects log files from the nodes using filters
>>-
>>
>>Some archives are created - general.tar.bz2 and logs-*
>>-
>>
>>checks are implemented to prevent filesystem filling due to log
>>collection, appropriate error shown.
>>-
>>
>>can be imported in other python scripts (ex.
>>https://github.com/f3flight/timmy-customtest) and used as a transport
>>and structure to access node parameters known to Fuel, run commands on
>>nodes, collect outputs, etc. with ease.
>>
>> ​
>>
>> Sincerely yours,
>> Aleksandr Dobdin
>> Senior Operations Engineer
>> Mirantis
>> ​Inc.​
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Dmitry Nikishov,
> Deployment Engineer,
> Mirantis, Inc.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Yours sincerely,
Dmitry Sutyagin
OpenStack Escalations Engineer
Mirantis, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] snapshot tool

[Dmitry Nikishov]

Hello,

I've got a couple of questions:
- What about this tool using non-root accounts to connect to OpenStack
nodes? Currently, it seems to assume that it always is going to use "root"
for SSH.
- Shouldn't it sanitize all sensitive information (user names, host names,
passwords, tokens, keys etc)?

Thanks.

On Tue, Apr 19, 2016 at 4:52 AM, Aleksandr Dobdin 
wrote:

> Hello team,
>
> I want to discuss the tool  that we
> have created for MOS as a replacement/alternative of shotgun.
>
>
>
>-
>
>The tool is based on
>https://etherpad.openstack.org/p/openstack-diagnostics
>-
>
>Should work fine on the following environments that were tested: 4.x,
>5.x, 6.x, 7.0, 8.0
>-
>
>Operates non-destructively.
>-
>
>Can be launched on any host within admin network, provided the fuel
>node IP is specified and access is possible to Fuel and other nodes via ssh
>from local system.
>-
>
>Parallel launch, only on the nodes that are 'online'. Some filters for
>nodes are also available.
>-
>
>Commands (from ./cmds directory) are separated according to roles
>(detected automatically) by the symlinks. Thus, the command list may depend
>on release, roles and OS. In addition, there can be some commands that run
>everywhere. There are also commands that are executed only on one node
>according to its role, using the first node of this type they
>encounter.
>-
>
>Modular: possible to create a special package that contains only
>certain required commands.
>-
>
>Collects log files from the nodes using filters
>-
>
>Some archives are created - general.tar.bz2 and logs-*
>-
>
>checks are implemented to prevent filesystem filling due to log
>collection, appropriate error shown.
>-
>
>can be imported in other python scripts (ex.
>https://github.com/f3flight/timmy-customtest) and used as a transport
>and structure to access node parameters known to Fuel, run commands on
>nodes, collect outputs, etc. with ease.
>
> ​
>
> Sincerely yours,
> Aleksandr Dobdin
> Senior Operations Engineer
> Mirantis
> ​Inc.​
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Dmitry Nikishov,
Deployment Engineer,
Mirantis, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] snapshot tool

[Aleksandr Dobdin]

Hello team,

I want to discuss the tool  that we have
created for MOS as a replacement/alternative of shotgun.



   -

   The tool is based on
   https://etherpad.openstack.org/p/openstack-diagnostics
   -

   Should work fine on the following environments that were tested: 4.x,
   5.x, 6.x, 7.0, 8.0
   -

   Operates non-destructively.
   -

   Can be launched on any host within admin network, provided the fuel node
   IP is specified and access is possible to Fuel and other nodes via ssh from
   local system.
   -

   Parallel launch, only on the nodes that are 'online'. Some filters for
   nodes are also available.
   -

   Commands (from ./cmds directory) are separated according to roles
   (detected automatically) by the symlinks. Thus, the command list may depend
   on release, roles and OS. In addition, there can be some commands that run
   everywhere. There are also commands that are executed only on one node
   according to its role, using the first node of this type they encounter.
   -

   Modular: possible to create a special package that contains only certain
   required commands.
   -

   Collects log files from the nodes using filters
   -

   Some archives are created - general.tar.bz2 and logs-*
   -

   checks are implemented to prevent filesystem filling due to log
   collection, appropriate error shown.
   -

   can be imported in other python scripts (ex.
   https://github.com/f3flight/timmy-customtest) and used as a transport
   and structure to access node parameters known to Fuel, run commands on
   nodes, collect outputs, etc. with ease.

​

Sincerely yours,
Aleksandr Dobdin
Senior Operations Engineer
Mirantis
​Inc.​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev