Also we thought about some kind of back-connected shell which allows customer to grant us full access to the master node instantly.
On Thu, Feb 27, 2014 at 8:28 PM, Sergey Yudin <[email protected]> wrote: > 2 Pavel Vaylov: > >Sergey, as I informed we have a solution for it, Andrey Ryabinin can > confirm (we bought subscription to some file sharing service). But the > solution is require to many manual steps from the Operation engineer. > Could you please clarify about that file sharing service? Is it suitable > for big files and private data? Maybe any documentation? > > 2 Matthew Mosesohn: > >Any log/snapshot submission system (like anonymous FTP upload) should accept > anonymous uploads (but not anonymous read/list) for user ease of use. > +1 for ftp, simple enoguht and suitable for our purpose. If there is no > solution exist yet, lets use ftp. > > >I believe anything extra beyond our current logging implementation should > involve manual command workarounds for data gathering. > Agreed, manual actions may be required, but even in that case i'm not sure > if it's desirable to involve customer into this process. It will be > easier if our manual workarounds will be shipped to the customer in a black > box, and will be executed as "some fuel diagnostic tool". > > And even if we will not take in according any custom manual workarounds we > still have to upgrade troubleshooting tool to be able to support customers > with old installation. > > 2 Vladimir Kozhukalov: > >Besides, our current solution has some problems with the ability to > gather diagnostic data from a number of nodes in parallel. At the moment it > is just one thread which runs one command on one remote node at each moment. > I've solved this issue in mine tool. > > >We also thought a bit about how to filter out sensitive data from > diagnostic snapshot > Well, it's countable number of objects which is sensetive for private > information, it only openstack configs in our case, i suppose. It is > possible to consider all of them, parse and cut passwords. It is not > implemented, but it seems possible. > > > >Anyway, please describe as much clearly as you can how you feel you can > integrate parts of your troubleshooting tool into Fuel diagnostic snapshot > tool. > It is just standalone python script right now, which is designed to be run > against fuel environment and generate single tarball with report. I'm not > familiar with fuel architerture enought to propose any solution on how to > integrate it into fuel, but it seems not so hard to be able via fuel UI > upload this tool for update, run it, and provide reports. > > > On Thu, Feb 27, 2014 at 6:01 PM, Vladimir Kozhukalov < > [email protected]> wrote: > >> Sergey, >> >> Great initiative. If you have thoughts about how we can improve Fuel >> "diagnostic snapshot" feature, let's discuss it in our maling list >> [email protected] or on our weekly IRC meeting >> https://wiki.openstack.org/wiki/Meetings/Fuel. >> >> Adding some commands to save their output is simple. One can add whatever >> they need into /etc/nailgun/settings.yaml file. Upgrading this tool on the >> fly is much more complicated improvement. Besides, our current solution has >> some problems with the ability to gather diagnostic data from a number of >> nodes in parallel. At the moment it is just one thread which runs one >> command on one remote node at each moment. >> >> We also thought a bit about how to filter out sensitive data from >> diagnostic snapshot, but we've not managed to find a suitable solution. At >> the moment all passwords are stored in a database as a plain text. So we >> tried to find all passwords wherever they appear and substitute them with a >> meaningless string. Because passwords maybe something like "1" or "admin", >> it is not so simple to filter them out without breaking other useful data. >> >> We also discussed the initiative which is about create shell wrapper for >> arbitrary command which would be able to save stdout and stderr into files >> or publish them on a paste service. >> >> Anyway, please describe as much clearly as you can how you feel you can >> integrate parts of your troubleshooting tool into Fuel diagnostic snapshot >> tool. >> >> >> Vladimir Kozhukalov >> >> >> On Thu, Feb 27, 2014 at 4:43 PM, Matthew Mosesohn <[email protected] >> > wrote: >> >>> Any log/snapshot submission system (like anonymous FTP upload) should >>> accept anonymous uploads (but not anonymous read/list) for user ease >>> of use. >>> >>> I believe anything extra beyond our current logging implementation >>> should involve manual command workarounds for data gathering. For >>> example, debugging glance-api with logs redirected for 5 minutes: >>> service openstack-glance-api stop >>> mkdir /root/glance-capture-logs >>> timeout 5m glance-api -v -d --log-file >>> /root/glance-capture-logs/glance-api.log >>> service openstack-glance-api start >>> >>> Then collector will pull all files out of /root, for example.. >>> >>> >>> >>> On Thu, Feb 27, 2014 at 4:36 PM, Pavel Vaylov <[email protected]> >>> wrote: >>> > 1. Absolutely need a way to upgrade/fix log collection process, not >>> only >>> > snapshot tool. For example this >>> https://bugs.launchpad.net/fuel/+bug/1284867 >>> > and probably new issues with logging. >>> > >>> > 2. Sergey, as I informed we have a solution for it, Andrey Ryabinin can >>> > confirm (we bought subscription to some file sharing service). But the >>> > solution is require to many manual steps from the Operation engineer. >>> > >>> > 3. Will test the tool on my Lab. >>> > >>> > >>> > >>> > >>> > On Thu, Feb 27, 2014 at 2:19 PM, Sergey Yudin <[email protected]> >>> wrote: >>> >> >>> >> Hi, guys. >>> >> Being part of TET team, i noticed that most difficult and long part of >>> >> troubleshotting is getting debug information from the customer, and i >>> want >>> >> improve this. >>> >> >>> >> I wanna discuss fuel snapshot tool, and related blueprint on >>> launchpad. >>> >> >>> https://blueprints.launchpad.net/fuel/+spec/fuel-snapshot-additional-info >>> >> >>> >> >>> >> From my point of view the tools not only has lack of information, as >>> >> mentioned in the blueprint, but also has lack of some base features >>> which >>> >> will make it usefull. >>> >> >>> >> 1. We need ability to upgrade this tool on fly, so if customer will >>> face >>> >> some issue support team will be able to send updated snapshot tool, >>> or even >>> >> fix exist tool for getting required information, and not rely on tool >>> which >>> >> was written years ago. >>> >> >>> >> 2. We need to think of storage for snapshots, which may be huge, so >>> will >>> >> not fit onto email, and also may contain private information, so will >>> not >>> >> fit for public storage. >>> >> >>> >> 3. There is tool written by me for gathering information from customer >>> >> installation, >>> >> >>> >> >>> https://docs.google.com/a/mirantis.com/document/d/1KyVuJcKy6lln4rI1d2kEZS63NU1ykswz0ib4MoM5bog/edit# >>> >> https://github.com/tsipa/troubleshooting >>> >> maybe we should use it. I'm agree to participate in integration it >>> into >>> >> fuel snapshot tool. >>> >> >>> >> Maybe i missed something and guys from support team has something >>> more to >>> >> say. >>> > >>> > >>> > >>> > >>> > -- >>> > Best regards >>> > >>> > Pavel Vaylov >>> > Operations Engineer >>> > Mirantis Service Desk >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "fuel-core-team" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an >>> > email to [email protected]. >>> > For more options, visit >>> > https://groups.google.com/a/mirantis.com/groups/opt_out. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "fuel-core-team" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit >>> https://groups.google.com/a/mirantis.com/groups/opt_out. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "fuel-core-team" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit > https://groups.google.com/a/mirantis.com/groups/opt_out. > -- Andrey Danin [email protected] skype: gcon.monolake
-- Mailing list: https://launchpad.net/~fuel-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~fuel-dev More help : https://help.launchpad.net/ListHelp
