Sorry Bin, this ended up in my spam somehow so I missed your update until a second ago.
Realistically, it's probably getting bottlenecked on the database. You can definitely try increasing the workers further (beyond 50) but I'm not sure how much it will help. A lot of the improvements in 3.14 are oriented around reducing our load on the database so it should help quite a bit. On Tue, Jun 22, 2021 at 12:35 PM Bin Li (BLOOMBERG/ 120 PARK) < bli...@bloomberg.net> wrote: > We will look into upgrade 3.7.3 to 3.14. > For now, I have updated number of worker a few times. We are having 50 > workers running. I no longer see the timed out messages but the TIME_WAIT > is still around 5k. > > # netstat -an | grep -i TIME_WAIT |grep 24816 | wc -l > 5473 > > Also notice the database connection is over 60. > => select count(*) from pg_stat_activity where usename = 'pulp'; > count > ------- > 63 > (1 row) > > Should I keep adding workers until the queue comes down? We still have > plenty of cpu and memory on the host. > > > From: bmbou...@redhat.com At: 06/22/21 12:01:30 UTC-4:00 > To: danny.sa...@konghq.com > Cc: Bin Li (BLOOMBERG/ 120 PARK ) <bli...@bloomberg.net>, > pulp-list@redhat.com > Subject: Re: [Pulp-list] Content server Performance > > > > On Tue, Jun 22, 2021 at 11:56 AM Danny Sauer <danny.sa...@konghq.com> > wrote: > >> You can certainly run multiple instances of the content server. It just >> needs a connection to the database and access to the storage. >> > Agreed, you could deploy additional content servers and have your > nginx/apache load balance them. > > >> Have you tuned the number of worker processes in Gunicorn? It defaults >> to 1, but should almost certainly be increased for any sort of volume. >> https://docs.gunicorn.org/en/stable/settings.html#worker-processes >> > Pulp changed the default gunicorn worker processes to 8 maybe a release or > two ago. See the `pulp_content_workers` variable in the installer here > https://pulp-installer.readthedocs.io/en/latest/roles/pulp_content/#role-variables > >> >> There are several moving pieces, but that's really all I had to touch >> here. >> >> --Danny >> > With pulpcore==3.14 there is a significant performance improvement being > reviewed now https://pulp.plan.io/issues/8805 . In addition to > resolving it with methods like ^, when 3.14 comes out (scheduled for June > 29th) it would be great if you could report on if the improvements helped > you. > >> >> On Tue, Jun 22, 2021 at 10:34 AM Bin Li (BLOOMBERG/ 120 PARK) < >> bli...@bloomberg.net> wrote: >> >>> We recently add more clients to use the pulp content server. The >>> processes run out the file descriptor first. We then increased both nginx >>> and pulp-content by creating a override.conf >>> /etc/systemd/system/pulpcore-content.service.d # cat override.conf >>> [Service] >>> LimitNOFILE=65536 >>> >>> and updated nginx.conf >>> # Gunicorn docs suggest this value. >>> worker_processes 1; >>> events { >>> worker_connections 10000; # increase if you have lots of clients >>> accept_mutex off; # set to 'on' if nginx worker_processes > 1 >>> } >>> >>> worker_rlimit_nofile 20000; >>> >>> >>> Now we are keep getting this error. >>> 2021/06/22 11:26:36 [error] 78373#0: *112823 upstream timed out (110: >>> Connection timed out) while connecting to upstream, client: >>> >>> It looks like pulp-content server cannot keep up with requests. Is there >>> anything we could do to increase the performance of the content server? >>> _______________________________________________ >>> Pulp-list mailing list >>> Pulp-list@redhat.com >>> https://listman.redhat.com/mailman/listinfo/pulp-list >> >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://listman.redhat.com/mailman/listinfo/pulp-list > > > _______________________________________________ > Pulp-list mailing list > Pulp-list@redhat.com > https://listman.redhat.com/mailman/listinfo/pulp-list
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://listman.redhat.com/mailman/listinfo/pulp-list
