I have an application I hope to eventually release to the wild that includes a
web-browser interface to one component. The HTTP server portion allows a user
to look at data, mostly event logs, and constructs a single page which consists
of some informational context with a variable amount of text embedded in it.
"Variable" means anything from one or two lines to literally tens of thousands
of lines. (The web interface allows you to specify a date range, among other
selection criteria, which can be years long.) The algorithm it uses, which I
did not write, is horrible for more than a thousand lines of data, getting
exponentially slower with size. I've seen perfectly reasonable requests take 20
minutes or more to render. I've attempted to fix it, with partial success, but
I think the best solution would be to break the output into "pages" with a
[more] button at the bottom. (Usually, when a user gets that much output, they
made a mistake in the request and don't really want it all. It is horrible to
type your search criteria, hit submit, and wait 20 minutes for results only to
find you made a stupid typo or unreasonable request and you have to do it
older. Worse than batch debugging Fortran on a CDC-6400 in 1973, using a card
reader and a line printer on a remote job entry station. (The 4800 baud leased
phone line was an insignificant factor in the turn-around!)
Anyway, I have no experience beyond writing "hello, world" and attempting to
debug or improve this program, but it sounds like the problem described here is
pretty analogous to my problem.
The whole thing (including the web server part, and the other 95% that I wrote)
is written in Python using Twisted for event processing, but the web server part
is the only part that uses HTTP and was written using the HTTP examples from the
old Twisted Network Programming Essentials (Fettig) book.
So I hope to follow along with this thread and learn something useful.
-- John Santos
On 12/12/2025 4:43 PM, [email protected] wrote:
On Dec 12, 2025, at 3:56 AM, Kirill Miazine <[email protected]> wrote:
Hi, list
The 65535 bytes limit on AMP values is a bit limiting sometimes. In my use
case I have a fleet of command runners, which receive requests to execute
some commands, execute commands and pass back commands' output. Especially
when fetching web pages, output will often exceed the AMP value limit. Fow
now I have been just cutting the output, but I'd like a better solution.
A simple, but somewhat ugly, approach will be to split output in parts and
send those as part1, [...], partN. It does not solve the issue fully, but
with N big enough, the workaround should be aceptable.
Is there an established pattern to use in such cases?
That's more or less the pattern. Usually if you need to send large values,
you also need to start taking backpressure, chunking, and multiplexing into
account, which is exactly what this somewhat arbitrary limit is meant to
encourage.
If you just need a higher limit for things that are on the order of 1-2
megabytes, where elaborate multiplexed streaming mechanisms are not worth
bothering with, you could look at an example like
https://www.evennia.com/docs/2.x/_modules/evennia/server/portal/amp.html . The
canonical example was originally at
https://code.launchpad.net/~glyph/+junk/amphacks but you can't view code on
launchpad any more.
If we were to improve AMP a bit, the length limit would be explicitly set,
rather than arbitrarily enforced by an implementation detail accident of the
protocol. I do want people to have to think about the length limit but
copy/pasting recipes like this and then forgetting it again is not really the
right way to encourage that.
-g
_______________________________________________
Twisted mailing list [email protected]
To unsubscribe send an email [email protected]
https://mail.python.org/mailman3/lists/twisted.python.org/
Message archived
athttps://mail.python.org/archives/list/[email protected]/message/ZUJOVGYIIYW25VYBOKITOYIFRJGCX6UW/
Code of Conduct:https://twisted.org/conduct
--
John Santos
Evans Griffiths & Hart, Inc.
_______________________________________________
Twisted mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/twisted.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/WQJTV7NZO4W6H7E7BUYGCDRY4RDGDF5Y/
Code of Conduct: https://twisted.org/conduct
