Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/19/2012 02:29 AM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 2:26 PM, Dmitri Pal wrote: >> On 07/18/2012 05:09 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 1:52 PM, Dmitri Pal wrote: On 07/18/2012 04:27 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: >> On 07/18/2012 03:45 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 12:28 PM, John Dennis >>> wrote: On 07/18/2012 02:59 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik > wrote: >> On 07/17/2012 11:43 PM, Stephen Ingram wrote: >> >> 8><-- >> >> > I'm beginning to think this is just the Web UI itself instead of > 389 > although it is really difficult to tell. I've poured over the > debug > logs and didn't see anything that caused me concern. > > It's certainly usable, but I just got really spoiled by the > unbelievable quickness of 2.1.3. When your release notes indicate > it > should be faster, what are you comparing it to? Maybe this only > happens with upgraded instances and not fresh installs. It is always possible something didn't get upgraded properly but I've done 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is faster we're always referring to the previous version (or versions). >>> Maybe I was just lucky with 2.1.3. On a first load it might take >>> some >>> time to load the "frame" as I call it. But the data would load >>> almost >>> instantaneously from there (certainly no more than 1 s) as you moved >>> from page to page. Here, even if I return to the same page, the >>> system >>> acts as if the data is begin fetched for the very first time as it >>> is >>> no faster than the first load. Maybe that is significant to the >>> problem? >> I think the culprit is Web UI paging capabilities introduced in 2.2. >> With >> lot of users, responses might grow in size. You can check their size >> and >> duration in browser developers tools. I suggest chrome/chromium - >> press >> F12 >> and choose 'network' tab. >> >> This new feature can't be disabled in configuration. To test if the >> slowdown >> is done by paging you can (at own risk) replace line >> /usr/share/ipa/ui/facet.js:538 >> >> that.pagination = spec.pagination === undefined ? true : >> spec.pagination; >> >> with: >> >> that.pagination = false; >> >> Note: It will break some other parts of the UI - so for testing only. > I've made the substitution in the code (was line 507 for me-do I have > a different version?). Looking at the time chart in Chrome I see that > the bulk of the time is for /ipa/session waiting. Would "waiting" mean > waiting for the directory server or memcached? Actually neither, it means waiting for a response from the web server (technically it's making an RPC call via HTTP Ajax). The RPC call needs to go through the web server, memcached, and typically will invoke one or more directory server queries, and run a bunch of Python to massage everything before the RPC returns with the result. It doesn't look like you've got much difference in times between with pagination on and pagination off. I don't know the pagination code but I suspect it's run after the RPC call returns so the RPC timing is not telling us much with respect to that. Waiting for up to 3 seconds for an RPC call does seem on the high side. Do you have a lot of LDAP data? >>> No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has >>> any significant amount of hosts in it. >>> But really, unless we get timing results for each component we're grasping at straws :-( >>> Understood. >>> >>> Steve >> Do you have a replica and does this replica behave the same? > No replica yet. I wanted to get the memory leak issue solved first. > All I have to compare to is the old 2.1.3 before. This one is much > slower. I just can't seem to figure out what's wrong. The upgrade > seemed to complete successfully and there were no errors in the log. > The only things I've found thus far (earlier in this thread) are the > unindexed entries (all hosts entries) that Rich seemed to think might > be slowing things up. As the slowness is on every
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Thu, 19 Jul 2012, John Dennis wrote: Rob may have already contacted you with this, but if not we would like to get more debugging information by have the server log what is occurring when it processes your requests. To do this you'll need to turn on the debug flag in the IPA configuration file /etc/ipa/default.conf, add a line that says: debug = True Then restart the server to pick up the new configuration. The information will be written to /var/log/httpd/error_log. We only need the contents of the log from when the server was restarted with debug logging enabled. For privacy reasons I suggest you send the contents of the log to one of the IPA team members directly in a private email, not to the public freeipa list. In addition we would like to see what's happening with krb5 communication under httpd processes. In order to obtain that tracing information you need to do following: 1. Add KRB5_TRACE=/tmp/http_krb5_trace.log to /etc/sysconfig/httpd 2. Restart httpd (or httpd.service in Fedora) 3. Now you need to create the file and chown it to apache's user so that httpd processes would be able to write to it: find out PID of any of httpd processes, doesn't matter which one touch /proc/$PID/cwd/tmp/http_krb5_trace.log chown apache /proc/$PID/cwd/tmp/http_krb5_trace.log 4. Now you can issue IPA commands and you'll get krb5 client tracing in /proc/$PID/cwd/tmp/http_krb5_trace.log The reason why (3) talks about PID of httpd process is because in Fedora, unlike in RHEL6.x, systemd is handling services startup and systemd confines httpd to a private /tmp. Using /proc/$PID/cwd/tmp is the easiest way to reach that private tmp. 5. Once finished and copied /proc/$PID/cwd/tmp/http_krb5_trace.log to an archive location, make sure to remove the file and its reference from /etc/sysconfig/httpd and restart the service. -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
Rob may have already contacted you with this, but if not we would like to get more debugging information by have the server log what is occurring when it processes your requests. To do this you'll need to turn on the debug flag in the IPA configuration file /etc/ipa/default.conf, add a line that says: debug = True Then restart the server to pick up the new configuration. The information will be written to /var/log/httpd/error_log. We only need the contents of the log from when the server was restarted with debug logging enabled. For privacy reasons I suggest you send the contents of the log to one of the IPA team members directly in a private email, not to the public freeipa list. Thanks! John -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 08:59 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik wrote: On 07/17/2012 11:43 PM, Stephen Ingram wrote: 8><-- I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. It is always possible something didn't get upgraded properly but I've done 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is faster we're always referring to the previous version (or versions). Maybe I was just lucky with 2.1.3. On a first load it might take some time to load the "frame" as I call it. But the data would load almost instantaneously from there (certainly no more than 1 s) as you moved from page to page. Here, even if I return to the same page, the system acts as if the data is begin fetched for the very first time as it is no faster than the first load. Maybe that is significant to the problem? I think the culprit is Web UI paging capabilities introduced in 2.2. With lot of users, responses might grow in size. You can check their size and duration in browser developers tools. I suggest chrome/chromium - press F12 and choose 'network' tab. This new feature can't be disabled in configuration. To test if the slowdown is done by paging you can (at own risk) replace line /usr/share/ipa/ui/facet.js:538 that.pagination = spec.pagination === undefined ? true : spec.pagination; with: that.pagination = false; Note: It will break some other parts of the UI - so for testing only. I've made the substitution in the code (was line 507 for me-do I have a different version?). I was looking at the top of FreeIPA 2.2 branch. RHEL version differs a bit. It shouldn't matter in this case though. Looking at the time chart in Chrome I see that the bulk of the time is for /ipa/session waiting. Would "waiting" mean waiting for the directory server or memcached? Basically all the stuff, which is needed for processing of the request. The pipeline is something like (I don't want to go into details): httpd -> mod_wsgi -> python -> memcache, dir server request and back. From what I see I think the problem is not on Web UI side. Most of the time is waiting for server response. I initially thought the problem lies in a large number of users (1000+). But in other post you mentioned that it is under 100. Hence this new paging feature shouldn't be a big problem. Here's a portion of the initial load of the Users page: The first 3 requests are inital load of web UI (not including .js files and such) - you can see they are the same in both cases. I don't see a login request so session is already established. json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.57s 1.47s 96ms (1.37s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 3.92s 2.95s 963ms (2.85s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.78s 2.94s 836ms (2.83s waiting) This one is user load: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.87s 1.71s (1.60s waiting) Now, with the pagination turned back on: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.58s 1.48s 100ms (1.38s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 4.05s 3.09s 964ms (2.98s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.84s 2.99s 855ms (2.88s waiting) Here you can see a change. With pagination turned on it should be two request. One to get primary keys (logins) and second to get users. The latter is missing in this list. With a lot of users the first response grows. With low number of users it is in fact smaller than with pagination turned off. json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.52s 1.51s (1.40s waiting) As I mentioned you are missing one request so it will add aprox. 1.5s. The second request is kinda a slowdown from IPA 2.1.4 but the main issue is still the long server processing time. Steve -- Petr Vobornik ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 2:26 PM, Dmitri Pal wrote: > On 07/18/2012 05:09 PM, Stephen Ingram wrote: >> On Wed, Jul 18, 2012 at 1:52 PM, Dmitri Pal wrote: >>> On 07/18/2012 04:27 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: > On 07/18/2012 03:45 PM, Stephen Ingram wrote: >> On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: >>> On 07/18/2012 02:59 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik wrote: > On 07/17/2012 11:43 PM, Stephen Ingram wrote: > > 8><-- > > I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. >>> >>> It is always possible something didn't get upgraded properly but >>> I've >>> done >>> 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say >>> something is >>> faster we're always referring to the previous version (or versions). >> Maybe I was just lucky with 2.1.3. On a first load it might take some >> time to load the "frame" as I call it. But the data would load almost >> instantaneously from there (certainly no more than 1 s) as you moved >> from page to page. Here, even if I return to the same page, the >> system >> acts as if the data is begin fetched for the very first time as it is >> no faster than the first load. Maybe that is significant to the >> problem? > I think the culprit is Web UI paging capabilities introduced in 2.2. > With > lot of users, responses might grow in size. You can check their size > and > duration in browser developers tools. I suggest chrome/chromium - > press > F12 > and choose 'network' tab. > > This new feature can't be disabled in configuration. To test if the > slowdown > is done by paging you can (at own risk) replace line > /usr/share/ipa/ui/facet.js:538 > > that.pagination = spec.pagination === undefined ? true : > spec.pagination; > > with: > > that.pagination = false; > > Note: It will break some other parts of the UI - so for testing only. I've made the substitution in the code (was line 507 for me-do I have a different version?). Looking at the time chart in Chrome I see that the bulk of the time is for /ipa/session waiting. Would "waiting" mean waiting for the directory server or memcached? >>> Actually neither, it means waiting for a response from the web server >>> (technically it's making an RPC call via HTTP Ajax). The RPC call needs >>> to >>> go through the web server, memcached, and typically will invoke one or >>> more >>> directory server queries, and run a bunch of Python to massage >>> everything >>> before the RPC returns with the result. >>> >>> It doesn't look like you've got much difference in times between with >>> pagination on and pagination off. I don't know the pagination code but I >>> suspect it's run after the RPC call returns so the RPC timing is not >>> telling >>> us much with respect to that. >>> >>> Waiting for up to 3 seconds for an RPC call does seem on the high side. >>> Do >>> you have a lot of LDAP data? >> No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has >> any significant amount of hosts in it. >> >>> But really, unless we get timing results for each component we're >>> grasping >>> at straws :-( >> Understood. >> >> Steve > Do you have a replica and does this replica behave the same? No replica yet. I wanted to get the memory leak issue solved first. All I have to compare to is the old 2.1.3 before. This one is much slower. I just can't seem to figure out what's wrong. The upgrade seemed to complete successfully and there were no errors in the log. The only things I've found thus far (earlier in this thread) are the unindexed entries (all hosts entries) that Rich seemed to think might be slowing things up. As the slowness is on every page, I wouldn't think that would be the problem. I wouldn't have said as much about this were it not for the promised faster speed mentioned in the release notes. It's comparable to the old 2.0 r
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 05:09 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 1:52 PM, Dmitri Pal wrote: >> On 07/18/2012 04:27 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: On 07/18/2012 03:45 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: >> On 07/18/2012 02:59 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik >>> wrote: On 07/17/2012 11:43 PM, Stephen Ingram wrote: 8><-- >>> I'm beginning to think this is just the Web UI itself instead of 389 >>> although it is really difficult to tell. I've poured over the debug >>> logs and didn't see anything that caused me concern. >>> >>> It's certainly usable, but I just got really spoiled by the >>> unbelievable quickness of 2.1.3. When your release notes indicate it >>> should be faster, what are you comparing it to? Maybe this only >>> happens with upgraded instances and not fresh installs. >> >> It is always possible something didn't get upgraded properly but I've >> done >> 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something >> is >> faster we're always referring to the previous version (or versions). > Maybe I was just lucky with 2.1.3. On a first load it might take some > time to load the "frame" as I call it. But the data would load almost > instantaneously from there (certainly no more than 1 s) as you moved > from page to page. Here, even if I return to the same page, the system > acts as if the data is begin fetched for the very first time as it is > no faster than the first load. Maybe that is significant to the > problem? I think the culprit is Web UI paging capabilities introduced in 2.2. With lot of users, responses might grow in size. You can check their size and duration in browser developers tools. I suggest chrome/chromium - press F12 and choose 'network' tab. This new feature can't be disabled in configuration. To test if the slowdown is done by paging you can (at own risk) replace line /usr/share/ipa/ui/facet.js:538 that.pagination = spec.pagination === undefined ? true : spec.pagination; with: that.pagination = false; Note: It will break some other parts of the UI - so for testing only. >>> I've made the substitution in the code (was line 507 for me-do I have >>> a different version?). Looking at the time chart in Chrome I see that >>> the bulk of the time is for /ipa/session waiting. Would "waiting" mean >>> waiting for the directory server or memcached? >> Actually neither, it means waiting for a response from the web server >> (technically it's making an RPC call via HTTP Ajax). The RPC call needs >> to >> go through the web server, memcached, and typically will invoke one or >> more >> directory server queries, and run a bunch of Python to massage everything >> before the RPC returns with the result. >> >> It doesn't look like you've got much difference in times between with >> pagination on and pagination off. I don't know the pagination code but I >> suspect it's run after the RPC call returns so the RPC timing is not >> telling >> us much with respect to that. >> >> Waiting for up to 3 seconds for an RPC call does seem on the high side. >> Do >> you have a lot of LDAP data? > No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has > any significant amount of hosts in it. > >> But really, unless we get timing results for each component we're >> grasping >> at straws :-( > Understood. > > Steve > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users Do you have a replica and does this replica behave the same? >>> No replica yet. I wanted to get the memory leak issue solved first. >>> All I have to compare to is the old 2.1.3 before. This one is much >>> slower. I just can't seem to figure out what's wrong. The upgrade >>> seemed to complete successfully and there were no errors in the log. >>> The only things I've found thus far (earlier in this thread) are the >>> unindexed entries (all hosts entries) that Rich seemed to think might >>> be slowing things up. As the slowness is on every page, I wouldn't >>> think that would be the problem. >>> >>> I wouldn't have said as much about this were it not for the promised >>> faster speed mentioned in the release notes. It's comparable to the >>> old 2.0 release candidates so I thought it might h
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 1:52 PM, Dmitri Pal wrote: > On 07/18/2012 04:27 PM, Stephen Ingram wrote: >> On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: >>> On 07/18/2012 03:45 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: > On 07/18/2012 02:59 PM, Stephen Ingram wrote: >> On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik >> wrote: >>> On 07/17/2012 11:43 PM, Stephen Ingram wrote: >>> >>> 8><-- >>> >>> >> I'm beginning to think this is just the Web UI itself instead of 389 >> although it is really difficult to tell. I've poured over the debug >> logs and didn't see anything that caused me concern. >> >> It's certainly usable, but I just got really spoiled by the >> unbelievable quickness of 2.1.3. When your release notes indicate it >> should be faster, what are you comparing it to? Maybe this only >> happens with upgraded instances and not fresh installs. > > > It is always possible something didn't get upgraded properly but I've > done > 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something > is > faster we're always referring to the previous version (or versions). Maybe I was just lucky with 2.1.3. On a first load it might take some time to load the "frame" as I call it. But the data would load almost instantaneously from there (certainly no more than 1 s) as you moved from page to page. Here, even if I return to the same page, the system acts as if the data is begin fetched for the very first time as it is no faster than the first load. Maybe that is significant to the problem? >>> >>> I think the culprit is Web UI paging capabilities introduced in 2.2. >>> With >>> lot of users, responses might grow in size. You can check their size and >>> duration in browser developers tools. I suggest chrome/chromium - press >>> F12 >>> and choose 'network' tab. >>> >>> This new feature can't be disabled in configuration. To test if the >>> slowdown >>> is done by paging you can (at own risk) replace line >>> /usr/share/ipa/ui/facet.js:538 >>> >>> that.pagination = spec.pagination === undefined ? true : >>> spec.pagination; >>> >>> with: >>> >>> that.pagination = false; >>> >>> Note: It will break some other parts of the UI - so for testing only. >> I've made the substitution in the code (was line 507 for me-do I have >> a different version?). Looking at the time chart in Chrome I see that >> the bulk of the time is for /ipa/session waiting. Would "waiting" mean >> waiting for the directory server or memcached? > Actually neither, it means waiting for a response from the web server > (technically it's making an RPC call via HTTP Ajax). The RPC call needs to > go through the web server, memcached, and typically will invoke one or > more > directory server queries, and run a bunch of Python to massage everything > before the RPC returns with the result. > > It doesn't look like you've got much difference in times between with > pagination on and pagination off. I don't know the pagination code but I > suspect it's run after the RPC call returns so the RPC timing is not > telling > us much with respect to that. > > Waiting for up to 3 seconds for an RPC call does seem on the high side. Do > you have a lot of LDAP data? No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has any significant amount of hosts in it. > But really, unless we get timing results for each component we're grasping > at straws :-( Understood. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users >>> Do you have a replica and does this replica behave the same? >> No replica yet. I wanted to get the memory leak issue solved first. >> All I have to compare to is the old 2.1.3 before. This one is much >> slower. I just can't seem to figure out what's wrong. The upgrade >> seemed to complete successfully and there were no errors in the log. >> The only things I've found thus far (earlier in this thread) are the >> unindexed entries (all hosts entries) that Rich seemed to think might >> be slowing things up. As the slowness is on every page, I wouldn't >> think that would be the problem. >> >> I wouldn't have said as much about this were it not for the promised >> faster speed mentioned in the release notes. It's comparable to the >> old 2.0 release candidates so I thought it might have been due to the >> complexity of the feature additions. >> >> Steve > Is the time correct on this system? Yes. HW clock is GMT and localtime is Paci
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 04:27 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: >> On 07/18/2012 03:45 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: On 07/18/2012 02:59 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik > wrote: >> On 07/17/2012 11:43 PM, Stephen Ingram wrote: >> >> 8><-- >> >> > I'm beginning to think this is just the Web UI itself instead of 389 > although it is really difficult to tell. I've poured over the debug > logs and didn't see anything that caused me concern. > > It's certainly usable, but I just got really spoiled by the > unbelievable quickness of 2.1.3. When your release notes indicate it > should be faster, what are you comparing it to? Maybe this only > happens with upgraded instances and not fresh installs. It is always possible something didn't get upgraded properly but I've done 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is faster we're always referring to the previous version (or versions). >>> >>> Maybe I was just lucky with 2.1.3. On a first load it might take some >>> time to load the "frame" as I call it. But the data would load almost >>> instantaneously from there (certainly no more than 1 s) as you moved >>> from page to page. Here, even if I return to the same page, the system >>> acts as if the data is begin fetched for the very first time as it is >>> no faster than the first load. Maybe that is significant to the >>> problem? >> >> I think the culprit is Web UI paging capabilities introduced in 2.2. With >> lot of users, responses might grow in size. You can check their size and >> duration in browser developers tools. I suggest chrome/chromium - press >> F12 >> and choose 'network' tab. >> >> This new feature can't be disabled in configuration. To test if the >> slowdown >> is done by paging you can (at own risk) replace line >> /usr/share/ipa/ui/facet.js:538 >> >> that.pagination = spec.pagination === undefined ? true : spec.pagination; >> >> with: >> >> that.pagination = false; >> >> Note: It will break some other parts of the UI - so for testing only. > I've made the substitution in the code (was line 507 for me-do I have > a different version?). Looking at the time chart in Chrome I see that > the bulk of the time is for /ipa/session waiting. Would "waiting" mean > waiting for the directory server or memcached? Actually neither, it means waiting for a response from the web server (technically it's making an RPC call via HTTP Ajax). The RPC call needs to go through the web server, memcached, and typically will invoke one or more directory server queries, and run a bunch of Python to massage everything before the RPC returns with the result. It doesn't look like you've got much difference in times between with pagination on and pagination off. I don't know the pagination code but I suspect it's run after the RPC call returns so the RPC timing is not telling us much with respect to that. Waiting for up to 3 seconds for an RPC call does seem on the high side. Do you have a lot of LDAP data? >>> No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has >>> any significant amount of hosts in it. >>> But really, unless we get timing results for each component we're grasping at straws :-( >>> Understood. >>> >>> Steve >>> >>> ___ >>> Freeipa-users mailing list >>> Freeipa-users@redhat.com >>> https://www.redhat.com/mailman/listinfo/freeipa-users >> Do you have a replica and does this replica behave the same? > No replica yet. I wanted to get the memory leak issue solved first. > All I have to compare to is the old 2.1.3 before. This one is much > slower. I just can't seem to figure out what's wrong. The upgrade > seemed to complete successfully and there were no errors in the log. > The only things I've found thus far (earlier in this thread) are the > unindexed entries (all hosts entries) that Rich seemed to think might > be slowing things up. As the slowness is on every page, I wouldn't > think that would be the problem. > > I wouldn't have said as much about this were it not for the promised > faster speed mentioned in the release notes. It's comparable to the > old 2.0 release candidates so I thought it might have been due to the > complexity of the feature additions. > > Steve Is the time correct on this system? -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freei
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 1:06 PM, Dmitri Pal wrote: > On 07/18/2012 03:45 PM, Stephen Ingram wrote: >> On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: >>> On 07/18/2012 02:59 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik wrote: > On 07/17/2012 11:43 PM, Stephen Ingram wrote: > > 8><-- > > I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. >>> >>> >>> >>> It is always possible something didn't get upgraded properly but I've >>> done >>> 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is >>> faster we're always referring to the previous version (or versions). >> >> >> Maybe I was just lucky with 2.1.3. On a first load it might take some >> time to load the "frame" as I call it. But the data would load almost >> instantaneously from there (certainly no more than 1 s) as you moved >> from page to page. Here, even if I return to the same page, the system >> acts as if the data is begin fetched for the very first time as it is >> no faster than the first load. Maybe that is significant to the >> problem? > > > I think the culprit is Web UI paging capabilities introduced in 2.2. With > lot of users, responses might grow in size. You can check their size and > duration in browser developers tools. I suggest chrome/chromium - press > F12 > and choose 'network' tab. > > This new feature can't be disabled in configuration. To test if the > slowdown > is done by paging you can (at own risk) replace line > /usr/share/ipa/ui/facet.js:538 > > that.pagination = spec.pagination === undefined ? true : spec.pagination; > > with: > > that.pagination = false; > > Note: It will break some other parts of the UI - so for testing only. I've made the substitution in the code (was line 507 for me-do I have a different version?). Looking at the time chart in Chrome I see that the bulk of the time is for /ipa/session waiting. Would "waiting" mean waiting for the directory server or memcached? >>> >>> Actually neither, it means waiting for a response from the web server >>> (technically it's making an RPC call via HTTP Ajax). The RPC call needs to >>> go through the web server, memcached, and typically will invoke one or more >>> directory server queries, and run a bunch of Python to massage everything >>> before the RPC returns with the result. >>> >>> It doesn't look like you've got much difference in times between with >>> pagination on and pagination off. I don't know the pagination code but I >>> suspect it's run after the RPC call returns so the RPC timing is not telling >>> us much with respect to that. >>> >>> Waiting for up to 3 seconds for an RPC call does seem on the high side. Do >>> you have a lot of LDAP data? >> No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has >> any significant amount of hosts in it. >> >>> But really, unless we get timing results for each component we're grasping >>> at straws :-( >> Understood. >> >> Steve >> >> ___ >> Freeipa-users mailing list >> Freeipa-users@redhat.com >> https://www.redhat.com/mailman/listinfo/freeipa-users > > Do you have a replica and does this replica behave the same? No replica yet. I wanted to get the memory leak issue solved first. All I have to compare to is the old 2.1.3 before. This one is much slower. I just can't seem to figure out what's wrong. The upgrade seemed to complete successfully and there were no errors in the log. The only things I've found thus far (earlier in this thread) are the unindexed entries (all hosts entries) that Rich seemed to think might be slowing things up. As the slowness is on every page, I wouldn't think that would be the problem. I wouldn't have said as much about this were it not for the promised faster speed mentioned in the release notes. It's comparable to the old 2.0 release candidates so I thought it might have been due to the complexity of the feature additions. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 03:45 PM, Stephen Ingram wrote: > On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: >> On 07/18/2012 02:59 PM, Stephen Ingram wrote: >>> On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik >>> wrote: On 07/17/2012 11:43 PM, Stephen Ingram wrote: 8><-- >>> I'm beginning to think this is just the Web UI itself instead of 389 >>> although it is really difficult to tell. I've poured over the debug >>> logs and didn't see anything that caused me concern. >>> >>> It's certainly usable, but I just got really spoiled by the >>> unbelievable quickness of 2.1.3. When your release notes indicate it >>> should be faster, what are you comparing it to? Maybe this only >>> happens with upgraded instances and not fresh installs. >> >> >> >> It is always possible something didn't get upgraded properly but I've >> done >> 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is >> faster we're always referring to the previous version (or versions). > > > Maybe I was just lucky with 2.1.3. On a first load it might take some > time to load the "frame" as I call it. But the data would load almost > instantaneously from there (certainly no more than 1 s) as you moved > from page to page. Here, even if I return to the same page, the system > acts as if the data is begin fetched for the very first time as it is > no faster than the first load. Maybe that is significant to the > problem? I think the culprit is Web UI paging capabilities introduced in 2.2. With lot of users, responses might grow in size. You can check their size and duration in browser developers tools. I suggest chrome/chromium - press F12 and choose 'network' tab. This new feature can't be disabled in configuration. To test if the slowdown is done by paging you can (at own risk) replace line /usr/share/ipa/ui/facet.js:538 that.pagination = spec.pagination === undefined ? true : spec.pagination; with: that.pagination = false; Note: It will break some other parts of the UI - so for testing only. >>> >>> I've made the substitution in the code (was line 507 for me-do I have >>> a different version?). Looking at the time chart in Chrome I see that >>> the bulk of the time is for /ipa/session waiting. Would "waiting" mean >>> waiting for the directory server or memcached? >> >> Actually neither, it means waiting for a response from the web server >> (technically it's making an RPC call via HTTP Ajax). The RPC call needs to >> go through the web server, memcached, and typically will invoke one or more >> directory server queries, and run a bunch of Python to massage everything >> before the RPC returns with the result. >> >> It doesn't look like you've got much difference in times between with >> pagination on and pagination off. I don't know the pagination code but I >> suspect it's run after the RPC call returns so the RPC timing is not telling >> us much with respect to that. >> >> Waiting for up to 3 seconds for an RPC call does seem on the high side. Do >> you have a lot of LDAP data? > No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has > any significant amount of hosts in it. > >> But really, unless we get timing results for each component we're grasping >> at straws :-( > Understood. > > Steve > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users Do you have a replica and does this replica behave the same? -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 10:59 AM, Dmitri Pal wrote: > On 07/18/2012 01:53 PM, Stephen Ingram wrote: >> On Tue, Jul 17, 2012 at 3:56 PM, John Dennis wrote: >>> On 07/17/2012 05:43 PM, Stephen Ingram wrote: >>> [ details of performance analysis snipped for brevity ] >>> I wonder if we shouldn't add some timing metrics to our code. As it is it's >>> very hard to know where time is being spent. >>> >>> When I wrote the session code I added some timestamps used for managing >>> session timeouts. It wouldn't be too hard to expand this to time how long it >>> takes a command to execute because it's evaluated for every command. >>> Combined with timestamping in the UI code we could get a reasonable idea of >>> where some bottlenecks lie (or don't). >> I've never used this before so I'm not sure how it would work, but it >> sounds great. It's really difficult to tell what's causing the issue >> when there are so many processes occurring. >> > > > While we are going with the technical digging let us also try to collect > the sufficient information about the problem. > > Here is some questions that would help us to reproduce the issue. > > 1) If the problem with every frame of just some specific UI pages? The frame seems to load quickly. It is the inside part that contains the data that is much slower. > Can you for example see IPA Configuration panel or log as a self service > user? Are those fast? I'm not sure what you mean by configuration panel, but if I login as admin or self-service user, they are both equally slow. > 2) Say it is users is so how many users do you have? Is it thousands? No, only 49 users at the moment. We're still adding people. There isn't a lot of data in the directory period--another reason I'm so surprised by the slowness. > Or may be it is a specific group? I notice that everyone is automatically subscribed to the ipausers group. Hasn't that been changed such that the subscription is no longer automatic? Maybe it is taking too long to enumerate that group? I can unsubscribe the users if needed. Our groups only contain 3 users on average. > We might need to reproduce the same setup and see what is going on. I'm more than willing to help in any way I can. I'm even considering pulling our old 2.1.3 system from backup, but it would be difficult as this on is in production now. I switched because of the memory leak in 2.1.3. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 12:28 PM, John Dennis wrote: > On 07/18/2012 02:59 PM, Stephen Ingram wrote: >> >> On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik >> wrote: >>> >>> On 07/17/2012 11:43 PM, Stephen Ingram wrote: >>> >>> 8><-- >>> >>> >> >> I'm beginning to think this is just the Web UI itself instead of 389 >> although it is really difficult to tell. I've poured over the debug >> logs and didn't see anything that caused me concern. >> >> It's certainly usable, but I just got really spoiled by the >> unbelievable quickness of 2.1.3. When your release notes indicate it >> should be faster, what are you comparing it to? Maybe this only >> happens with upgraded instances and not fresh installs. > > > > > It is always possible something didn't get upgraded properly but I've > done > 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is > faster we're always referring to the previous version (or versions). Maybe I was just lucky with 2.1.3. On a first load it might take some time to load the "frame" as I call it. But the data would load almost instantaneously from there (certainly no more than 1 s) as you moved from page to page. Here, even if I return to the same page, the system acts as if the data is begin fetched for the very first time as it is no faster than the first load. Maybe that is significant to the problem? >>> >>> >>> >>> I think the culprit is Web UI paging capabilities introduced in 2.2. With >>> lot of users, responses might grow in size. You can check their size and >>> duration in browser developers tools. I suggest chrome/chromium - press >>> F12 >>> and choose 'network' tab. >>> >>> This new feature can't be disabled in configuration. To test if the >>> slowdown >>> is done by paging you can (at own risk) replace line >>> /usr/share/ipa/ui/facet.js:538 >>> >>> that.pagination = spec.pagination === undefined ? true : spec.pagination; >>> >>> with: >>> >>> that.pagination = false; >>> >>> Note: It will break some other parts of the UI - so for testing only. >> >> >> I've made the substitution in the code (was line 507 for me-do I have >> a different version?). Looking at the time chart in Chrome I see that >> the bulk of the time is for /ipa/session waiting. Would "waiting" mean >> waiting for the directory server or memcached? > > > Actually neither, it means waiting for a response from the web server > (technically it's making an RPC call via HTTP Ajax). The RPC call needs to > go through the web server, memcached, and typically will invoke one or more > directory server queries, and run a bunch of Python to massage everything > before the RPC returns with the result. > > It doesn't look like you've got much difference in times between with > pagination on and pagination off. I don't know the pagination code but I > suspect it's run after the RPC call returns so the RPC timing is not telling > us much with respect to that. > > Waiting for up to 3 seconds for an RPC call does seem on the high side. Do > you have a lot of LDAP data? No. 49 users, 17 hosts, 25 services, 6 DNS zones, only 1 of which has any significant amount of hosts in it. > But really, unless we get timing results for each component we're grasping > at straws :-( Understood. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 02:59 PM, Stephen Ingram wrote: On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik wrote: On 07/17/2012 11:43 PM, Stephen Ingram wrote: 8><-- I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. It is always possible something didn't get upgraded properly but I've done 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is faster we're always referring to the previous version (or versions). Maybe I was just lucky with 2.1.3. On a first load it might take some time to load the "frame" as I call it. But the data would load almost instantaneously from there (certainly no more than 1 s) as you moved from page to page. Here, even if I return to the same page, the system acts as if the data is begin fetched for the very first time as it is no faster than the first load. Maybe that is significant to the problem? I think the culprit is Web UI paging capabilities introduced in 2.2. With lot of users, responses might grow in size. You can check their size and duration in browser developers tools. I suggest chrome/chromium - press F12 and choose 'network' tab. This new feature can't be disabled in configuration. To test if the slowdown is done by paging you can (at own risk) replace line /usr/share/ipa/ui/facet.js:538 that.pagination = spec.pagination === undefined ? true : spec.pagination; with: that.pagination = false; Note: It will break some other parts of the UI - so for testing only. I've made the substitution in the code (was line 507 for me-do I have a different version?). Looking at the time chart in Chrome I see that the bulk of the time is for /ipa/session waiting. Would "waiting" mean waiting for the directory server or memcached? Actually neither, it means waiting for a response from the web server (technically it's making an RPC call via HTTP Ajax). The RPC call needs to go through the web server, memcached, and typically will invoke one or more directory server queries, and run a bunch of Python to massage everything before the RPC returns with the result. It doesn't look like you've got much difference in times between with pagination on and pagination off. I don't know the pagination code but I suspect it's run after the RPC call returns so the RPC timing is not telling us much with respect to that. Waiting for up to 3 seconds for an RPC call does seem on the high side. Do you have a lot of LDAP data? But really, unless we get timing results for each component we're grasping at straws :-( Here's a portion of the initial load of the Users page: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.57s 1.47s 96ms (1.37s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 3.92s 2.95s 963ms (2.85s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.78s 2.94s 836ms (2.83s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.87s 1.71s (1.60s waiting) Now, with the pagination turned back on: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.58s 1.48s 100ms (1.38s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 4.05s 3.09s 964ms (2.98s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.84s 2.99s 855ms (2.88s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.52s 1.51s (1.40s waiting) Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Wed, Jul 18, 2012 at 6:45 AM, Petr Vobornik wrote: > On 07/17/2012 11:43 PM, Stephen Ingram wrote: > > 8><-- > > I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. >>> >>> >>> >>> It is always possible something didn't get upgraded properly but I've >>> done >>> 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is >>> faster we're always referring to the previous version (or versions). >> >> >> Maybe I was just lucky with 2.1.3. On a first load it might take some >> time to load the "frame" as I call it. But the data would load almost >> instantaneously from there (certainly no more than 1 s) as you moved >> from page to page. Here, even if I return to the same page, the system >> acts as if the data is begin fetched for the very first time as it is >> no faster than the first load. Maybe that is significant to the >> problem? > > > I think the culprit is Web UI paging capabilities introduced in 2.2. With > lot of users, responses might grow in size. You can check their size and > duration in browser developers tools. I suggest chrome/chromium - press F12 > and choose 'network' tab. > > This new feature can't be disabled in configuration. To test if the slowdown > is done by paging you can (at own risk) replace line > /usr/share/ipa/ui/facet.js:538 > > that.pagination = spec.pagination === undefined ? true : spec.pagination; > > with: > > that.pagination = false; > > Note: It will break some other parts of the UI - so for testing only. I've made the substitution in the code (was line 507 for me-do I have a different version?). Looking at the time chart in Chrome I see that the bulk of the time is for /ipa/session waiting. Would "waiting" mean waiting for the directory server or memcached? Here's a portion of the initial load of the Users page: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.57s 1.47s 96ms (1.37s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 3.92s 2.95s 963ms (2.85s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.78s 2.94s 836ms (2.83s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.87s 1.71s (1.60s waiting) Now, with the pagination turned back on: json/ipa/session POST 200 Success application/json jquery.js:7365 Script 33.94KB 33.10KB 1.58s 1.48s 100ms (1.38s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 568.09KB 564.36KB 4.05s 3.09s 964ms (2.98s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 556.94KB 553.40KB 3.84s 2.99s 855ms (2.88s waiting) json/ipa/session POST 200 Success application/json jquery.js:7365 Script 46.93KB 46.38KB 1.52s 1.51s (1.40s waiting) Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/18/2012 01:53 PM, Stephen Ingram wrote: > On Tue, Jul 17, 2012 at 3:56 PM, John Dennis wrote: >> On 07/17/2012 05:43 PM, Stephen Ingram wrote: >> >>> [ details of performance analysis snipped for brevity ] >> I wonder if we shouldn't add some timing metrics to our code. As it is it's >> very hard to know where time is being spent. >> >> When I wrote the session code I added some timestamps used for managing >> session timeouts. It wouldn't be too hard to expand this to time how long it >> takes a command to execute because it's evaluated for every command. >> Combined with timestamping in the UI code we could get a reasonable idea of >> where some bottlenecks lie (or don't). > I've never used this before so I'm not sure how it would work, but it > sounds great. It's really difficult to tell what's causing the issue > when there are so many processes occurring. > While we are going with the technical digging let us also try to collect the sufficient information about the problem. Here is some questions that would help us to reproduce the issue. 1) If the problem with every frame of just some specific UI pages? Can you for example see IPA Configuration panel or log as a self service user? Are those fast? 2) Say it is users is so how many users do you have? Is it thousands? Or may be it is a specific group? We might need to reproduce the same setup and see what is going on. > Steve > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Tue, Jul 17, 2012 at 3:56 PM, John Dennis wrote: > On 07/17/2012 05:43 PM, Stephen Ingram wrote: > >> [ details of performance analysis snipped for brevity ] > > I wonder if we shouldn't add some timing metrics to our code. As it is it's > very hard to know where time is being spent. > > When I wrote the session code I added some timestamps used for managing > session timeouts. It wouldn't be too hard to expand this to time how long it > takes a command to execute because it's evaluated for every command. > Combined with timestamping in the UI code we could get a reasonable idea of > where some bottlenecks lie (or don't). I've never used this before so I'm not sure how it would work, but it sounds great. It's really difficult to tell what's causing the issue when there are so many processes occurring. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/17/2012 11:43 PM, Stephen Ingram wrote: 8><-- I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to tell. I've poured over the debug logs and didn't see anything that caused me concern. It's certainly usable, but I just got really spoiled by the unbelievable quickness of 2.1.3. When your release notes indicate it should be faster, what are you comparing it to? Maybe this only happens with upgraded instances and not fresh installs. It is always possible something didn't get upgraded properly but I've done 2.1.3 -> 2.2.0 upgrades and haven't seen this. When we say something is faster we're always referring to the previous version (or versions). Maybe I was just lucky with 2.1.3. On a first load it might take some time to load the "frame" as I call it. But the data would load almost instantaneously from there (certainly no more than 1 s) as you moved from page to page. Here, even if I return to the same page, the system acts as if the data is begin fetched for the very first time as it is no faster than the first load. Maybe that is significant to the problem? I think the culprit is Web UI paging capabilities introduced in 2.2. With lot of users, responses might grow in size. You can check their size and duration in browser developers tools. I suggest chrome/chromium - press F12 and choose 'network' tab. This new feature can't be disabled in configuration. To test if the slowdown is done by paging you can (at own risk) replace line /usr/share/ipa/ui/facet.js:538 that.pagination = spec.pagination === undefined ? true : spec.pagination; with: that.pagination = false; Note: It will break some other parts of the UI - so for testing only. Steve -- Petr Vobornik ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/17/2012 05:43 PM, Stephen Ingram wrote: > [ details of performance analysis snipped for brevity ] I wonder if we shouldn't add some timing metrics to our code. As it is it's very hard to know where time is being spent. When I wrote the session code I added some timestamps used for managing session timeouts. It wouldn't be too hard to expand this to time how long it takes a command to execute because it's evaluated for every command. Combined with timestamping in the UI code we could get a reasonable idea of where some bottlenecks lie (or don't). -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Tue, Jul 17, 2012 at 2:01 PM, Rob Crittenden wrote: > Stephen Ingram wrote: >> >> On Mon, Jul 16, 2012 at 12:23 PM, Rob Crittenden >> wrote: >>> >>> Stephen Ingram wrote: On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson wrote: > > > On 07/16/2012 11:48 AM, Stephen Ingram wrote: >> >> >> >> On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson >> wrote: >>> >>> >>> >>> On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: > > > > Stephen Ingram wrote: >> >> >> >> On Thu, Jul 12, 2012 at 2:59 PM, Steven >> Jones >> wrote: >>> >>> >>> >>> Hi, >>> >>> I had huge memory issues pre 6.3, now its low and flatSounds >>> like >>> you >>> have an issue somewhere. My normal cpu use is a few hundred >>> mhzbut >>> when >>> "something" goes wrong such as replication failing that >>> climbs...ditto >>> memory use >> >> >> >> >> Yes, I saw your conversation with Rich on this list about that. >> And, >> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is >> still >> having issues. It was an upgrade from 2.1.3, but the upgrade >> seemed >> to >> complete without issue. I'm also not even doing replication yet so >> I'm >> not sure why memory is so high. Web interface is much slower too >> so >> perhaps something else is wrong. > > > > > Can you tell where it is being slow? Does it seem related to > retrieving > data > from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. > You might check your 389-ds access logs and look for searches with > notes=U. > Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? >>> >>> >>> >>> Try running logconv.pl >> >> >> >> Nice! I'm guessing that notes=U are unindexed searches then. I have 34 >> over the last 24 hours so I'm not sure this would be causing the issue >> as the slowness persists through every click. > > > > Yeah, I would expect to see a lot more than 34 if that were the cause. > > Can you post the search filters that are unindexed? Sure, here's a partial list (sanitized): filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) attrs="fqdn" filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" All the rest are the same, just with different hosts. >> I've traced the >> unindexed searches back to the time of Web UI access and they don't >> match. I also don't see any other obvious errors when running >> logconv.pl. >> >> One strange thing I have noticed is that the 389 server logs seem to >> update in "spurts". If I'm tailing the logs while I access a Web UI >> page, there is nothing, then a couple of seconds later, I see the logs >> quickly scroll with new entires. Has this always been the case? I >> don't seem to remember this before. > > > > Yes. The 389 access log is buffered, for performance reasons. Just thought it might be relevant. I'm not sure what is causing the extreme slowness. I've also shut off memcached and tried without it with no discernible difference. The directory seems to be handling the load of external queries just fine, although I'm not sure I've solved the memory issue--I'm still testing with the compat plugin disabled to see if I can stop the memory creep. Maybe it's something in the code of the Web UI itself as its even slow when changing from page to page of users and hosts. >>> >>> >>> >>> Shutting of memcached is just going to make things even slower. >> >> >> I
Re: [Freeipa-users] 2.20 dirsrv memory usage
Stephen Ingram wrote: On Mon, Jul 16, 2012 at 12:23 PM, Rob Crittenden wrote: Stephen Ingram wrote: On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson wrote: On 07/16/2012 11:48 AM, Stephen Ingram wrote: On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson wrote: On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Can you tell where it is being slow? Does it seem related to retrieving data from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. You might check your 389-ds access logs and look for searches with notes=U. Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? Try running logconv.pl Nice! I'm guessing that notes=U are unindexed searches then. I have 34 over the last 24 hours so I'm not sure this would be causing the issue as the slowness persists through every click. Yeah, I would expect to see a lot more than 34 if that were the cause. Can you post the search filters that are unindexed? Sure, here's a partial list (sanitized): filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) attrs="fqdn" filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" All the rest are the same, just with different hosts. I've traced the unindexed searches back to the time of Web UI access and they don't match. I also don't see any other obvious errors when running logconv.pl. One strange thing I have noticed is that the 389 server logs seem to update in "spurts". If I'm tailing the logs while I access a Web UI page, there is nothing, then a couple of seconds later, I see the logs quickly scroll with new entires. Has this always been the case? I don't seem to remember this before. Yes. The 389 access log is buffered, for performance reasons. Just thought it might be relevant. I'm not sure what is causing the extreme slowness. I've also shut off memcached and tried without it with no discernible difference. The directory seems to be handling the load of external queries just fine, although I'm not sure I've solved the memory issue--I'm still testing with the compat plugin disabled to see if I can stop the memory creep. Maybe it's something in the code of the Web UI itself as its even slow when changing from page to page of users and hosts. Shutting of memcached is just going to make things even slower. I really didn't see much difference so I turned it back on right away. Things you can try on a quiet system: 1. Create /etc/ipa/server.conf: [global] debug=True Restart Apache Use the UI to do a few things. Verify in the logs that the session cache is being used. Yes, it is. It's interesting, 2.2 is slower such that you can see the frame load, and then the loading symbol spins below in the display area for a few seconds while that loads up. Before, with 2.1.3, you really couldn't distinguish the two as they loaded so quickly. A lot of what gets loaded a just big javascript files. I wonder if there is a DNS problem, that would explain the slowness. The javascript and much of the UI is completely unprotected by anything. 2. Check your browser configuration. You don't need delegation-uris set any more. Having it set might mask other problems (you still need negotiation-auth.trusted-uris set). I forgot about this. I changed it, completely cleared the browser cache and accessed without any noticeable difference. 3. Watch what URI is being used in the Apache access log. It should be /ipa/session/json. Check. this is where it lands. I'm beginning to think this is just the Web UI itself instead of 389 although it is really difficult to
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Mon, Jul 16, 2012 at 12:23 PM, Rob Crittenden wrote: > Stephen Ingram wrote: >> >> On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson >> wrote: >>> >>> On 07/16/2012 11:48 AM, Stephen Ingram wrote: On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson wrote: > > > On 07/16/2012 10:19 AM, Stephen Ingram wrote: >> >> >> On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden >> wrote: >>> >>> >>> Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: > > > Hi, > > I had huge memory issues pre 6.3, now its low and flatSounds > like > you > have an issue somewhere. My normal cpu use is a few hundred > mhzbut > when > "something" goes wrong such as replication failing that > climbs...ditto > memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. >>> >>> >>> >>> Can you tell where it is being slow? Does it seem related to >>> retrieving >>> data >>> from LDAP? >> >> >> I'm not really sure yet what is causing the slowness. I have the same >> number of directory entries as before the upgrade. It was very quick >> with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 >> days--without a doubt much, much slower. >> >>> You might check your 389-ds access logs and look for searches with >>> notes=U. >>> Perhaps you are missing an index. >> >> >> Yes there are lots of notes=U. What does this mean? Was something >> missed in the upgrade script? > > > Try running logconv.pl Nice! I'm guessing that notes=U are unindexed searches then. I have 34 over the last 24 hours so I'm not sure this would be causing the issue as the slowness persists through every click. >>> >>> >>> Yeah, I would expect to see a lot more than 34 if that were the cause. >>> >>> Can you post the search filters that are unindexed? >> >> >> Sure, here's a partial list (sanitized): >> >> >> filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) >> attrs="fqdn" >> >> filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" >> attrs="fqdn" >> >> filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" >> attrs="fqdn" >> >> filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" >> attrs="fqdn" >> >> filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" >> attrs="fqdn" >> >> All the rest are the same, just with different hosts. >> I've traced the unindexed searches back to the time of Web UI access and they don't match. I also don't see any other obvious errors when running logconv.pl. One strange thing I have noticed is that the 389 server logs seem to update in "spurts". If I'm tailing the logs while I access a Web UI page, there is nothing, then a couple of seconds later, I see the logs quickly scroll with new entires. Has this always been the case? I don't seem to remember this before. >>> >>> >>> Yes. The 389 access log is buffered, for performance reasons. >> >> >> Just thought it might be relevant. I'm not sure what is causing the >> extreme slowness. I've also shut off memcached and tried without it >> with no discernible difference. The directory seems to be handling the >> load of external queries just fine, although I'm not sure I've solved >> the memory issue--I'm still testing with the compat plugin disabled to >> see if I can stop the memory creep. Maybe it's something in the code >> of the Web UI itself as its even slow when changing from page to page >> of users and hosts. > > > Shutting of memcached is just going to make things even slower. I really didn't see much difference so I turned it back on right away. > Things you can try on a quiet system: > > 1. Create /etc/ipa/server.conf: > > [global] > debug=True > > Restart Apache > > Use the UI to do a few things. Verify in the logs that the session cache is > being used. Yes, it is. It's interesting, 2.2 is slower such that you can see the frame load, and then the loading symbol spins below in the display area for a few seconds while that loads up. Before, with 2.1.3, you really couldn't distinguish the two as they loaded so quickly. > 2. Check your browser configuration. You
Re: [Freeipa-users] 2.20 dirsrv memory usage
Stephen Ingram wrote: On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson wrote: On 07/16/2012 11:48 AM, Stephen Ingram wrote: On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson wrote: On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Can you tell where it is being slow? Does it seem related to retrieving data from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. You might check your 389-ds access logs and look for searches with notes=U. Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? Try running logconv.pl Nice! I'm guessing that notes=U are unindexed searches then. I have 34 over the last 24 hours so I'm not sure this would be causing the issue as the slowness persists through every click. Yeah, I would expect to see a lot more than 34 if that were the cause. Can you post the search filters that are unindexed? Sure, here's a partial list (sanitized): filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) attrs="fqdn" filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" All the rest are the same, just with different hosts. I've traced the unindexed searches back to the time of Web UI access and they don't match. I also don't see any other obvious errors when running logconv.pl. One strange thing I have noticed is that the 389 server logs seem to update in "spurts". If I'm tailing the logs while I access a Web UI page, there is nothing, then a couple of seconds later, I see the logs quickly scroll with new entires. Has this always been the case? I don't seem to remember this before. Yes. The 389 access log is buffered, for performance reasons. Just thought it might be relevant. I'm not sure what is causing the extreme slowness. I've also shut off memcached and tried without it with no discernible difference. The directory seems to be handling the load of external queries just fine, although I'm not sure I've solved the memory issue--I'm still testing with the compat plugin disabled to see if I can stop the memory creep. Maybe it's something in the code of the Web UI itself as its even slow when changing from page to page of users and hosts. Shutting of memcached is just going to make things even slower. Things you can try on a quiet system: 1. Create /etc/ipa/server.conf: [global] debug=True Restart Apache Use the UI to do a few things. Verify in the logs that the session cache is being used. 2. Check your browser configuration. You don't need delegation-uris set any more. Having it set might mask other problems (you still need negotiation-auth.trusted-uris set). 3. Watch what URI is being used in the Apache access log. It should be /ipa/session/json. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Mon, 2012-07-16 at 12:11 -0700, Stephen Ingram wrote: > On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson wrote: > > On 07/16/2012 11:48 AM, Stephen Ingram wrote: > >> > >> On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson > >> wrote: > >>> > >>> On 07/16/2012 10:19 AM, Stephen Ingram wrote: > > On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden > wrote: > > > > Stephen Ingram wrote: > >> > >> On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones > >> wrote: > >>> > >>> Hi, > >>> > >>> I had huge memory issues pre 6.3, now its low and flatSounds like > >>> you > >>> have an issue somewhere. My normal cpu use is a few hundred > >>> mhzbut > >>> when > >>> "something" goes wrong such as replication failing that > >>> climbs...ditto > >>> memory use > >> > >> > >> Yes, I saw your conversation with Rich on this list about that. And, > >> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still > >> having issues. It was an upgrade from 2.1.3, but the upgrade seemed to > >> complete without issue. I'm also not even doing replication yet so I'm > >> not sure why memory is so high. Web interface is much slower too so > >> perhaps something else is wrong. > > > > > > Can you tell where it is being slow? Does it seem related to retrieving > > data > > from LDAP? > > I'm not really sure yet what is causing the slowness. I have the same > number of directory entries as before the upgrade. It was very quick > with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 > days--without a doubt much, much slower. > > > You might check your 389-ds access logs and look for searches with > > notes=U. > > Perhaps you are missing an index. > > Yes there are lots of notes=U. What does this mean? Was something > missed in the upgrade script? > >>> > >>> Try running logconv.pl > >> > >> Nice! I'm guessing that notes=U are unindexed searches then. I have 34 > >> over the last 24 hours so I'm not sure this would be causing the issue > >> as the slowness persists through every click. > > > > Yeah, I would expect to see a lot more than 34 if that were the cause. > > > > Can you post the search filters that are unindexed? > > Sure, here's a partial list (sanitized): > > filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) > attrs="fqdn" > filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" > attrs="fqdn" > filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" > attrs="fqdn" > filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" > attrs="fqdn" > filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" > attrs="fqdn" > > All the rest are the same, just with different hosts. > > >> I've traced the > >> unindexed searches back to the time of Web UI access and they don't > >> match. I also don't see any other obvious errors when running > >> logconv.pl. > >> > >> One strange thing I have noticed is that the 389 server logs seem to > >> update in "spurts". If I'm tailing the logs while I access a Web UI > >> page, there is nothing, then a couple of seconds later, I see the logs > >> quickly scroll with new entires. Has this always been the case? I > >> don't seem to remember this before. > > > > Yes. The 389 access log is buffered, for performance reasons. > > Just thought it might be relevant. I'm not sure what is causing the > extreme slowness. I've also shut off memcached and tried without it > with no discernible difference. The directory seems to be handling the > load of external queries just fine, although I'm not sure I've solved > the memory issue--I'm still testing with the compat plugin disabled to > see if I can stop the memory creep. Maybe it's something in the code > of the Web UI itself as its even slow when changing from page to page > of users and hosts. Looks like the symptoms of not using session cookies. Do you see constant activity getting tickets for ldap/ipa.server.fqdn in the krb5kdc.log ? Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Mon, Jul 16, 2012 at 11:34 AM, Rich Megginson wrote: > On 07/16/2012 11:48 AM, Stephen Ingram wrote: >> >> On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson >> wrote: >>> >>> On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: > > Stephen Ingram wrote: >> >> On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones >> wrote: >>> >>> Hi, >>> >>> I had huge memory issues pre 6.3, now its low and flatSounds like >>> you >>> have an issue somewhere. My normal cpu use is a few hundred >>> mhzbut >>> when >>> "something" goes wrong such as replication failing that >>> climbs...ditto >>> memory use >> >> >> Yes, I saw your conversation with Rich on this list about that. And, >> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still >> having issues. It was an upgrade from 2.1.3, but the upgrade seemed to >> complete without issue. I'm also not even doing replication yet so I'm >> not sure why memory is so high. Web interface is much slower too so >> perhaps something else is wrong. > > > Can you tell where it is being slow? Does it seem related to retrieving > data > from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. > You might check your 389-ds access logs and look for searches with > notes=U. > Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? >>> >>> Try running logconv.pl >> >> Nice! I'm guessing that notes=U are unindexed searches then. I have 34 >> over the last 24 hours so I'm not sure this would be causing the issue >> as the slowness persists through every click. > > Yeah, I would expect to see a lot more than 34 if that were the cause. > > Can you post the search filters that are unindexed? Sure, here's a partial list (sanitized): filter="(managedBy=fqdn=ec2-x.x.x.us-west-2.compute.amazonaws.com,cn=computers,cn=accounts,dc=example,dc=com) attrs="fqdn" filter="(managedBy=fqdn=imap.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=imap2.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" filter="(managedBy=fqdn=ipa1.example.com,cn=computers,cn=accounts,dc=example,dc=com)" attrs="fqdn" All the rest are the same, just with different hosts. >> I've traced the >> unindexed searches back to the time of Web UI access and they don't >> match. I also don't see any other obvious errors when running >> logconv.pl. >> >> One strange thing I have noticed is that the 389 server logs seem to >> update in "spurts". If I'm tailing the logs while I access a Web UI >> page, there is nothing, then a couple of seconds later, I see the logs >> quickly scroll with new entires. Has this always been the case? I >> don't seem to remember this before. > > Yes. The 389 access log is buffered, for performance reasons. Just thought it might be relevant. I'm not sure what is causing the extreme slowness. I've also shut off memcached and tried without it with no discernible difference. The directory seems to be handling the load of external queries just fine, although I'm not sure I've solved the memory issue--I'm still testing with the compat plugin disabled to see if I can stop the memory creep. Maybe it's something in the code of the Web UI itself as its even slow when changing from page to page of users and hosts. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/16/2012 11:48 AM, Stephen Ingram wrote: On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson wrote: On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Can you tell where it is being slow? Does it seem related to retrieving data from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. You might check your 389-ds access logs and look for searches with notes=U. Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? Try running logconv.pl Nice! I'm guessing that notes=U are unindexed searches then. I have 34 over the last 24 hours so I'm not sure this would be causing the issue as the slowness persists through every click. Yeah, I would expect to see a lot more than 34 if that were the cause. Can you post the search filters that are unindexed? I've traced the unindexed searches back to the time of Web UI access and they don't match. I also don't see any other obvious errors when running logconv.pl. One strange thing I have noticed is that the 389 server logs seem to update in "spurts". If I'm tailing the logs while I access a Web UI page, there is nothing, then a couple of seconds later, I see the logs quickly scroll with new entires. Has this always been the case? I don't seem to remember this before. Yes. The 389 access log is buffered, for performance reasons. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Mon, Jul 16, 2012 at 9:35 AM, Rich Megginson wrote: > On 07/16/2012 10:19 AM, Stephen Ingram wrote: >> >> On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden >> wrote: >>> >>> Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: > > Hi, > > I had huge memory issues pre 6.3, now its low and flatSounds like > you > have an issue somewhere. My normal cpu use is a few hundred mhzbut > when > "something" goes wrong such as replication failing that climbs...ditto > memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. >>> >>> >>> Can you tell where it is being slow? Does it seem related to retrieving >>> data >>> from LDAP? >> >> I'm not really sure yet what is causing the slowness. I have the same >> number of directory entries as before the upgrade. It was very quick >> with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 >> days--without a doubt much, much slower. >> >>> You might check your 389-ds access logs and look for searches with >>> notes=U. >>> Perhaps you are missing an index. >> >> Yes there are lots of notes=U. What does this mean? Was something >> missed in the upgrade script? > > Try running logconv.pl Nice! I'm guessing that notes=U are unindexed searches then. I have 34 over the last 24 hours so I'm not sure this would be causing the issue as the slowness persists through every click. I've traced the unindexed searches back to the time of Web UI access and they don't match. I also don't see any other obvious errors when running logconv.pl. One strange thing I have noticed is that the 389 server logs seem to update in "spurts". If I'm tailing the logs while I access a Web UI page, there is nothing, then a couple of seconds later, I see the logs quickly scroll with new entires. Has this always been the case? I don't seem to remember this before. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/16/2012 10:19 AM, Stephen Ingram wrote: On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Can you tell where it is being slow? Does it seem related to retrieving data from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. You might check your 389-ds access logs and look for searches with notes=U. Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? Try running logconv.pl Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Fri, Jul 13, 2012 at 6:14 AM, Rob Crittenden wrote: > Stephen Ingram wrote: >> >> On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones >> wrote: >>> >>> Hi, >>> >>> I had huge memory issues pre 6.3, now its low and flatSounds like you >>> have an issue somewhere. My normal cpu use is a few hundred mhzbut when >>> "something" goes wrong such as replication failing that climbs...ditto >>> memory use >> >> >> Yes, I saw your conversation with Rich on this list about that. And, >> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still >> having issues. It was an upgrade from 2.1.3, but the upgrade seemed to >> complete without issue. I'm also not even doing replication yet so I'm >> not sure why memory is so high. Web interface is much slower too so >> perhaps something else is wrong. > > > Can you tell where it is being slow? Does it seem related to retrieving data > from LDAP? I'm not really sure yet what is causing the slowness. I have the same number of directory entries as before the upgrade. It was very quick with 2.1.3, but once I upgraded, I felt like I was back to the pre-2.0 days--without a doubt much, much slower. > You might check your 389-ds access logs and look for searches with notes=U. > Perhaps you are missing an index. Yes there are lots of notes=U. What does this mean? Was something missed in the upgrade script? Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Can you tell where it is being slow? Does it seem related to retrieving data from LDAP? You might check your 389-ds access logs and look for searches with notes=U. Perhaps you are missing an index. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/12/2012 06:55 PM, Stephen Ingram wrote: > On Thu, Jul 12, 2012 at 3:41 PM, Dmitri Pal wrote: >> On 07/12/2012 06:19 PM, Stephen Ingram wrote: >>> On Thu, Jul 12, 2012 at 3:10 PM, Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: > Hi, > > I had huge memory issues pre 6.3, now its low and flatSounds like you > have an issue somewhere. My normal cpu use is a few hundred mhzbut > when "something" goes wrong such as replication failing that > climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. >>> Oops, I meant Rob, not Rich. >> Do you use any things exposed via compat tree? >> Do you have a lot of modifications that affect the data that is exposed >> via this tree? >> I suspect that the leak is somewhere there. >> >> Try turning off the things that you do not use if there are any. > I only query cn=users,cn=accounts,dc=example,dc=com and > cn=groups,cn=accounts,dc=example,dc=com containers for mail info and > use for Kerberos auth. There are only very infrequent mods to those > trees previously mentioned via the Web UI. Almost all activity is > reads, but lots of them for the mail servers (validating users, etc.). > I plan to use replication and DNS, but not using now. From this, I > don't think I'm using the compat tree. Would turning it off help > anyway? > > Steve Please use documented tools to disable it. Do not do it manually via LDAP. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/12/2012 06:55 PM, Stephen Ingram wrote: > On Thu, Jul 12, 2012 at 3:41 PM, Dmitri Pal wrote: >> On 07/12/2012 06:19 PM, Stephen Ingram wrote: >>> On Thu, Jul 12, 2012 at 3:10 PM, Stephen Ingram wrote: On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: > Hi, > > I had huge memory issues pre 6.3, now its low and flatSounds like you > have an issue somewhere. My normal cpu use is a few hundred mhzbut > when "something" goes wrong such as replication failing that > climbs...ditto memory use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. >>> Oops, I meant Rob, not Rich. >> Do you use any things exposed via compat tree? >> Do you have a lot of modifications that affect the data that is exposed >> via this tree? >> I suspect that the leak is somewhere there. >> >> Try turning off the things that you do not use if there are any. > I only query cn=users,cn=accounts,dc=example,dc=com and > cn=groups,cn=accounts,dc=example,dc=com containers for mail info and > use for Kerberos auth. There are only very infrequent mods to those > trees previously mentioned via the Web UI. Almost all activity is > reads, but lots of them for the mail servers (validating users, etc.). > I plan to use replication and DNS, but not using now. From this, I > don't think I'm using the compat tree. Would turning it off help > anyway? > > Steve It is worth a try. There is a known slow minor leak in the compat plugin. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Thu, Jul 12, 2012 at 3:41 PM, Dmitri Pal wrote: > On 07/12/2012 06:19 PM, Stephen Ingram wrote: >> On Thu, Jul 12, 2012 at 3:10 PM, Stephen Ingram wrote: >>> On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones >>> wrote: Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use >>> Yes, I saw your conversation with Rich on this list about that. And, >>> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still >>> having issues. It was an upgrade from 2.1.3, but the upgrade seemed to >>> complete without issue. I'm also not even doing replication yet so I'm >>> not sure why memory is so high. Web interface is much slower too so >>> perhaps something else is wrong. >> Oops, I meant Rob, not Rich. > > Do you use any things exposed via compat tree? > Do you have a lot of modifications that affect the data that is exposed > via this tree? > I suspect that the leak is somewhere there. > > Try turning off the things that you do not use if there are any. I only query cn=users,cn=accounts,dc=example,dc=com and cn=groups,cn=accounts,dc=example,dc=com containers for mail info and use for Kerberos auth. There are only very infrequent mods to those trees previously mentioned via the Web UI. Almost all activity is reads, but lots of them for the mail servers (validating users, etc.). I plan to use replication and DNS, but not using now. From this, I don't think I'm using the compat tree. Would turning it off help anyway? Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On 07/12/2012 06:19 PM, Stephen Ingram wrote: > On Thu, Jul 12, 2012 at 3:10 PM, Stephen Ingram wrote: >> On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: >>> Hi, >>> >>> I had huge memory issues pre 6.3, now its low and flatSounds like you >>> have an issue somewhere. My normal cpu use is a few hundred mhzbut when >>> "something" goes wrong such as replication failing that climbs...ditto >>> memory use >> Yes, I saw your conversation with Rich on this list about that. And, >> yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still >> having issues. It was an upgrade from 2.1.3, but the upgrade seemed to >> complete without issue. I'm also not even doing replication yet so I'm >> not sure why memory is so high. Web interface is much slower too so >> perhaps something else is wrong. > Oops, I meant Rob, not Rich. Do you use any things exposed via compat tree? Do you have a lot of modifications that affect the data that is exposed via this tree? I suspect that the leak is somewhere there. Try turning off the things that you do not use if there are any. > Steve > > ___ > Freeipa-users mailing list > Freeipa-users@redhat.com > https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Thu, Jul 12, 2012 at 3:10 PM, Stephen Ingram wrote: > On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: >> Hi, >> >> I had huge memory issues pre 6.3, now its low and flatSounds like you >> have an issue somewhere. My normal cpu use is a few hundred mhzbut when >> "something" goes wrong such as replication failing that climbs...ditto >> memory use > > Yes, I saw your conversation with Rich on this list about that. And, > yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still > having issues. It was an upgrade from 2.1.3, but the upgrade seemed to > complete without issue. I'm also not even doing replication yet so I'm > not sure why memory is so high. Web interface is much slower too so > perhaps something else is wrong. Oops, I meant Rob, not Rich. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
On Thu, Jul 12, 2012 at 2:59 PM, Steven Jones wrote: > Hi, > > I had huge memory issues pre 6.3, now its low and flatSounds like you > have an issue somewhere. My normal cpu use is a few hundred mhzbut when > "something" goes wrong such as replication failing that climbs...ditto memory > use Yes, I saw your conversation with Rich on this list about that. And, yes, 6.2 (2.1.3) was bad for me too. I'm not sure why 2.2.0 is still having issues. It was an upgrade from 2.1.3, but the upgrade seemed to complete without issue. I'm also not even doing replication yet so I'm not sure why memory is so high. Web interface is much slower too so perhaps something else is wrong. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 2.20 dirsrv memory usage
Hi, I had huge memory issues pre 6.3, now its low and flatSounds like you have an issue somewhere. My normal cpu use is a few hundred mhzbut when "something" goes wrong such as replication failing that climbs...ditto memory use regards Steven Jones Technical Specialist - Linux RHCE Victoria University, Wellington, NZ 0064 4 463 6272 From: freeipa-users-boun...@redhat.com [freeipa-users-boun...@redhat.com] on behalf of Stephen Ingram [sbing...@gmail.com] Sent: Friday, 13 July 2012 9:29 a.m. To: freeipa-users Subject: [Freeipa-users] 2.20 dirsrv memory usage I was previously using 2.1.4 and know that there was a substantial memory leak in the directory server. After upgrading to 2.20, I notice that although overall memory usage seems higher, the "creep" upwards is not as quick. Although memory still tends to trend upward leaving me to worry that dirsrv will crash when it runs out of memory. I've checked the entrycachehitratio and it is 99. I also then checked the size of id2entry.db4 and found it to be 1024000. So I then checked nssldap-cachesize and found it to be 10485760. According to what I've read on the list, this seems about right. Is there anything else I can check? This is a pretty small directory, but gets quite a bit of activity from serving mail configuration in addition to authentication. However, I can't imagine that it would consume 1.5GB and keep climbing in memory usage. Steve ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users