Re: [ovirt-users] audit_log table performance tuning

[Marina Kalinin]

- Original Message -
> On 15.04.2016 00:12, Marina Kalinin wrote:
> > Hi,
> > 
> > Any suggestions or maybe already available features in the pipeline for
> > tuning the database, and specifically the audit_log table?
> > 
> > The problem today is that with multiple applications accessing the engine
> > through the RestAPI, especially deployments with CloudForms, create huge
> > amount of login records in the audit_table. Which, in turns, consumes most
> > of the available memory on the machine running the engine and the database
> > and results in a terrible performance of engine and inaccessible Web UI.
> > 
> > The solution today is to delete those records from the table [1]:
> > => delete from audit_log where message like '%logged%';
> > 
> > 
> > Are there any current tunings we can apply to the database?
> > And if not - do we have any RFEs on limiting the records entered to the
> > database or a way to delete/filter those records somehow from the WebUI?
> > All I could find was RFE#1120659 [2], but it does not describe the exact
> > issue.
> > 
> > 
> 
> Hi,
> 
> I remember filing a BZ about this topic some years ago.
> 
> I will mail it tomorrow to this thread as I had this exact issue, as a
> user of the rest api (without any persistent authentication).
Sven, Did you find anything?

I still cannot put my head around the right solution for this case. So I would 
appreciate to see your bug.
> 
> the answer, as I recall (I haven't access to this BZ atm), was to simply
> truncate the event log, which is far from a "solution" at all.
> 
> kind regards
> 
> Sven
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 

-- 
--
 mku
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] audit_log table performance tuning

[Sven Kieske]

On 15.04.2016 00:12, Marina Kalinin wrote:
> Hi,
> 
> Any suggestions or maybe already available features in the pipeline for 
> tuning the database, and specifically the audit_log table? 
> 
> The problem today is that with multiple applications accessing the engine 
> through the RestAPI, especially deployments with CloudForms, create huge 
> amount of login records in the audit_table. Which, in turns, consumes most of 
> the available memory on the machine running the engine and the database and 
> results in a terrible performance of engine and inaccessible Web UI.
> 
> The solution today is to delete those records from the table [1]:
> => delete from audit_log where message like '%logged%';
> 
> 
> Are there any current tunings we can apply to the database? 
> And if not - do we have any RFEs on limiting the records entered to the 
> database or a way to delete/filter those records somehow from the WebUI?
> All I could find was RFE#1120659 [2], but it does not describe the exact 
> issue.
> 
> 

Hi,

I remember filing a BZ about this topic some years ago.

I will mail it tomorrow to this thread as I had this exact issue, as a
user of the rest api (without any persistent authentication).

the answer, as I recall (I haven't access to this BZ atm), was to simply
truncate the event log, which is far from a "solution" at all.

kind regards

Sven



signature.asc
Description: OpenPGP digital signature
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] audit_log table performance tuning

[Juan Hernández]

On 04/23/2016 07:17 AM, Oved Ourfali wrote:
> 
> On Apr 23, 2016 7:46 AM, "Marina Kalinin"  > wrote:
>>
>> Hi all,
>>
>> So far I created this solution for immediate remedy:
>> https://access.redhat.com/solutions/721423
>>
>> I created this general RFE, that would help in our situation:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1329793
>>
>> However, this RFE is not all what I have in my mind.
> 
> I also don't think we should have such tool as mentioned in the RFE.
> 
>> I am thinking if there is anyway we can limit the number of identical
> records in audit_log?
> 
> Identical records might imply some issue, so I don't think they should
> be limited. What we can add configuration for is maybe the frequency in
> which the data is purged, which is 30 days currently, afair.
> 
>> Or, as Oved suggested, something to do with RestAPI and CFME to reduce
> the amount of logging?
>> BTW, does current version of CFME already contains this feature:
>> http://old.ovirt.org/Features/RESTSessionManagement
>> ?
> 
> CFME uses the session. However, I think that we don't reuse the session
> among unrelated requests.
> Not 100% sure about that.
> 
> Juan?
> 

CFME does use the persistent authentication mechanism. It creates new
sessions (which triggers the audit log message) in the following situations:

1. When CFME starts it creates a new session to poll RHEV-M events. This
session is reused as long as CFME is running.

2. When a refresh of the inventory is performed. Each refresh creates a
new session.

3. When performing actions like start or shutdown a VM. Each of these
actions creates a new session.

So unless there is a high level of activity in the environment (starting
virtual machines, stopping them, etc) there shouldn't be a high amount
of activity in the logs.

In systems where high activity is detected I'd suggest to configure the
RHEVM-M web server log file so that we will have a clear idea of what is
creating sessions. This can be achieved adding the following line to the
/etc/httpd/conf.d/ssl.conf file:

  CustomLog logs/api_log "%h %t \"%r\" User-Agent=\"%{User-Agent}i\"
JSESSIONID=\"%{JSESSIONID}C\" Prefer=\"%{Prefer}i\" %>s"

Then restart the web server:

  # service httpd restart

The content of that "api_log" file would help us what is really sending
requests that trigger the login/logout audit messages.

Note that the additional log file consumes resources, both CPU and disk
space, so it should be disabled once the relevant information is captured.

>>
>> Thank you,
>> Marina.
>>
>> 
>>>
>>>
>>>
>>> On Sun, Apr 17, 2016 at 9:33 AM, Oved Ourfali  > wrote:

 Juan - we should try to reduce this number consumed by CFME, if
> possible.
 CC-ing Eli for DB related tips.
>>>
>>>
>>> Reminds me of https://gerrit.ovirt.org/#/c/55743/ .
>>
>> This commit is great, however not relevant for my case, since
> event_notification_hist is empty.
>>>
>>> Y.


 On Fri, Apr 15, 2016 at 1:12 AM, Marina Kalinin  > wrote:
>
> Hi,
>
> Any suggestions or maybe already available features in the pipeline
> for tuning the database, and specifically the audit_log table?
>
> The problem today is that with multiple applications accessing the
> engine through the RestAPI, especially deployments with CloudForms,
> create huge amount of login records in the audit_table. Which, in turns,
> consumes most of the available memory on the machine running the engine
> and the database and results in a terrible performance of engine and
> inaccessible Web UI.
>
> The solution today is to delete those records from the table [1]:
> => delete from audit_log where message like '%logged%';
>
>
> Are there any current tunings we can apply to the database?
> And if not - do we have any RFEs on limiting the records entered to
> the database or a way to delete/filter those records somehow from the WebUI?
> All I could find was RFE#1120659 [2], but it does not describe the
> exact issue.
>
>
> --
> Thanks,
> Marina.
>
>
> [1] https://access.redhat.com/solutions/2110011
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users
>
>


 ___
 Users mailing list
 Users@ovirt.org 
 http://lists.ovirt.org/mailman/listinfo/users

>>>
>>
>>
>>
>> --
>> --
>>  mku
> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 


-- 
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta

Re: [ovirt-users] audit_log table performance tuning

[Marina Kalinin]

Hi all, 

So far I created this solution for immediate remedy: 
https://access.redhat.com/solutions/721423 

I created this general RFE, that would help in our situation: 
https://bugzilla.redhat.com/show_bug.cgi?id=1329793 

However, this RFE is not all what I have in my mind. 
I am thinking if there is anyway we can limit the number of identical records 
in audit_log? 
Or, as Oved suggested, something to do with RestAPI and CFME to reduce the 
amount of logging? 
BTW, does current version of CFME already contains this feature: 
http://old.ovirt.org/Features/RESTSessionManagement 
? 

Thank you, 
Marina. 

- Original Message -

> On Sun, Apr 17, 2016 at 9:33 AM, Oved Ourfali < oourf...@redhat.com > wrote:

> > Juan - we should try to reduce this number consumed by CFME, if possible.
> 
> > CC-ing Eli for DB related tips.
> 

> Reminds me of https://gerrit.ovirt.org/#/c/55743/ .

This commit is great, however not relevant for my case, since 
event_notification_hist is empty. 

> Y.

> > On Fri, Apr 15, 2016 at 1:12 AM, Marina Kalinin < mkali...@redhat.com >
> > wrote:
> 

> > > Hi,
> > 
> 

> > > Any suggestions or maybe already available features in the pipeline for
> > > tuning the database, and specifically the audit_log table?
> > 
> 

> > > The problem today is that with multiple applications accessing the engine
> > > through the RestAPI, especially deployments with CloudForms, create huge
> > > amount of login records in the audit_table. Which, in turns, consumes
> > > most
> > > of the available memory on the machine running the engine and the
> > > database
> > > and results in a terrible performance of engine and inaccessible Web UI.
> > 
> 

> > > The solution today is to delete those records from the table [1]:
> > 
> 
> > > => delete from audit_log where message like '%logged%';
> > 
> 

> > > Are there any current tunings we can apply to the database?
> > 
> 
> > > And if not - do we have any RFEs on limiting the records entered to the
> > > database or a way to delete/filter those records somehow from the WebUI?
> > 
> 
> > > All I could find was RFE#1120659 [2], but it does not describe the exact
> > > issue.
> > 
> 

> > > --
> > 
> 
> > > Thanks,
> > 
> 
> > > Marina.
> > 
> 

> > > [1] https://access.redhat.com/solutions/2110011
> > 
> 
> > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
> > 
> 
> > > ___
> > 
> 
> > > Users mailing list
> > 
> 
> > > Users@ovirt.org
> > 
> 
> > > http://lists.ovirt.org/mailman/listinfo/users
> > 
> 

> > ___
> 
> > Users mailing list
> 
> > Users@ovirt.org
> 
> > http://lists.ovirt.org/mailman/listinfo/users
> 

-- 
-- 
mku 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] audit_log table performance tuning

[Oved Ourfali]

On Apr 23, 2016 7:46 AM, "Marina Kalinin"  wrote:
>
> Hi all,
>
> So far I created this solution for immediate remedy:
> https://access.redhat.com/solutions/721423
>
> I created this general RFE, that would help in our situation:
> https://bugzilla.redhat.com/show_bug.cgi?id=1329793
>
> However, this RFE is not all what I have in my mind.

I also don't think we should have such tool as mentioned in the RFE.

> I am thinking if there is anyway we can limit the number of identical
records in audit_log?

Identical records might imply some issue, so I don't think they should be
limited. What we can add configuration for is maybe the frequency in which
the data is purged, which is 30 days currently, afair.

> Or, as Oved suggested, something to do with RestAPI and CFME to reduce
the amount of logging?
> BTW, does current version of CFME already contains this feature:
> http://old.ovirt.org/Features/RESTSessionManagement
> ?

CFME uses the session. However, I think that we don't reuse the session
among unrelated requests.
Not 100% sure about that.

Juan?

>
> Thank you,
> Marina.
>
> 
>>
>>
>>
>> On Sun, Apr 17, 2016 at 9:33 AM, Oved Ourfali 
wrote:
>>>
>>> Juan - we should try to reduce this number consumed by CFME, if
possible.
>>> CC-ing Eli for DB related tips.
>>
>>
>> Reminds me of https://gerrit.ovirt.org/#/c/55743/ .
>
> This commit is great, however not relevant for my case, since
event_notification_hist is empty.
>>
>> Y.
>>>
>>>
>>> On Fri, Apr 15, 2016 at 1:12 AM, Marina Kalinin 
wrote:

 Hi,

 Any suggestions or maybe already available features in the pipeline
for tuning the database, and specifically the audit_log table?

 The problem today is that with multiple applications accessing the
engine through the RestAPI, especially deployments with CloudForms, create
huge amount of login records in the audit_table. Which, in turns, consumes
most of the available memory on the machine running the engine and the
database and results in a terrible performance of engine and inaccessible
Web UI.

 The solution today is to delete those records from the table [1]:
 => delete from audit_log where message like '%logged%';


 Are there any current tunings we can apply to the database?
 And if not - do we have any RFEs on limiting the records entered to
the database or a way to delete/filter those records somehow from the WebUI?
 All I could find was RFE#1120659 [2], but it does not describe the
exact issue.


 --
 Thanks,
 Marina.


 [1] https://access.redhat.com/solutions/2110011
 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users


>>>
>>>
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
>
>
> --
> --
>  mku
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] audit_log table performance tuning

[Yaniv Kaul]

On Sun, Apr 17, 2016 at 9:33 AM, Oved Ourfali  wrote:

> Juan - we should try to reduce this number consumed by CFME, if possible.
> CC-ing Eli for DB related tips.
>

Reminds me of https://gerrit.ovirt.org/#/c/55743/ .
Y.


>
> On Fri, Apr 15, 2016 at 1:12 AM, Marina Kalinin 
> wrote:
>
>> Hi,
>>
>> Any suggestions or maybe already available features in the pipeline for
>> tuning the database, and specifically the audit_log table?
>>
>> The problem today is that with multiple applications accessing the engine
>> through the RestAPI, especially deployments with CloudForms, create huge
>> amount of login records in the audit_table. Which, in turns, consumes most
>> of the available memory on the machine running the engine and the database
>> and results in a terrible performance of engine and inaccessible Web UI.
>>
>> The solution today is to delete those records from the table [1]:
>> => delete from audit_log where message like '%logged%';
>>
>>
>> Are there any current tunings we can apply to the database?
>> And if not - do we have any RFEs on limiting the records entered to the
>> database or a way to delete/filter those records somehow from the WebUI?
>> All I could find was RFE#1120659 [2], but it does not describe the exact
>> issue.
>>
>>
>> --
>> Thanks,
>> Marina.
>>
>>
>> [1] https://access.redhat.com/solutions/2110011
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] audit_log table performance tuning

[Oved Ourfali]

Juan - we should try to reduce this number consumed by CFME, if possible.
CC-ing Eli for DB related tips.

On Fri, Apr 15, 2016 at 1:12 AM, Marina Kalinin  wrote:

> Hi,
>
> Any suggestions or maybe already available features in the pipeline for
> tuning the database, and specifically the audit_log table?
>
> The problem today is that with multiple applications accessing the engine
> through the RestAPI, especially deployments with CloudForms, create huge
> amount of login records in the audit_table. Which, in turns, consumes most
> of the available memory on the machine running the engine and the database
> and results in a terrible performance of engine and inaccessible Web UI.
>
> The solution today is to delete those records from the table [1]:
> => delete from audit_log where message like '%logged%';
>
>
> Are there any current tunings we can apply to the database?
> And if not - do we have any RFEs on limiting the records entered to the
> database or a way to delete/filter those records somehow from the WebUI?
> All I could find was RFE#1120659 [2], but it does not describe the exact
> issue.
>
>
> --
> Thanks,
> Marina.
>
>
> [1] https://access.redhat.com/solutions/2110011
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] audit_log table performance tuning

[Marina Kalinin]

Hi,

Any suggestions or maybe already available features in the pipeline for tuning 
the database, and specifically the audit_log table? 

The problem today is that with multiple applications accessing the engine 
through the RestAPI, especially deployments with CloudForms, create huge amount 
of login records in the audit_table. Which, in turns, consumes most of the 
available memory on the machine running the engine and the database and results 
in a terrible performance of engine and inaccessible Web UI.

The solution today is to delete those records from the table [1]:
=> delete from audit_log where message like '%logged%';


Are there any current tunings we can apply to the database? 
And if not - do we have any RFEs on limiting the records entered to the 
database or a way to delete/filter those records somehow from the WebUI?
All I could find was RFE#1120659 [2], but it does not describe the exact issue.


-- 
Thanks,
Marina.


[1] https://access.redhat.com/solutions/2110011
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1120659
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users