> The feature is currently used only a few times a year, and I could see
that going to a couple times a week at most
A common problem is that if someone hits reload, or has an internet
connectivity issue, you end up with one worker stalled and the second
worker handling this. You can very easily grind a site to halt with this.
I learned this the hard way. I now shard all admin routes for apps using
the technique mentioned above, so the expensive operations of a few users
do not affect the performance of the public routes. I sometimes layer in
S3 as well.
I've never had those issues with S3.
On Saturday, September 13, 2025 at 6:37:07 PM UTC-4 Mike Orr wrote:
> On Sat, Sep 13, 2025 at 2:30 PM Jonathan Vanasco
> wrote:
> > I recommend against this approach, as it does not scale well. The issue
> is this requires a dedicated Pyramid worker (thread or process). IMHO, it's
> best to offload stuff like this onto something like Nginx/Apache/HAProxy
> that can handle concurrency and streaming better, while freeing up the
> worker. A common approach is to store some metadata about the generated
> file and stash it in a database, then have a cronjob (or similar) read that
> database and delete expired files after 24 hours.
>
> Good to know, but this is a low-volume site with a restricted
> userbase. The feature is currently used only a few times a year, and I
> could see that going to a couple times a week at most. But while it's
> used infrequently, some of its use cases are essential. It replaces an
> offline script that will be harder to support in the new IT
> infrastructure. And I needed to implement it quickly to fold it into a
> release this month.
>
> > > Personally in cases like this I upload the file to S3 and then
> redirect the user to the signed URL. S3 lifecycle can be configured to
> delete the file after a day, which keeps storage costs minimal.
>
> I made a prototype of that for another feature, serving attachment
> files interactively in the site. But one of the sysadmins recommended
> against this approach, saying using S3 to serve directly to users was
> prone to delays.
>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/pylons-discuss/2b8658c8-e669-4d20-8126-bf5b22a0d7c7n%40googlegroups.com.
