rrwright opened a new pull request, #744:
URL: https://github.com/apache/pekko-http/pull/744
# Problem
The current SSE (Server-Sent Events) client implementation fails its stream
every time a message arrives over the size limits. When a Pekko SSE client
connects to a Pekko SSE server who serves a message larger than the size limit,
the client fails the stream, tries to reconnect and resume at the same time
(oversized) message and fails again. The client falls into an infinite
connection retry loop. The oversized message error is swallowed by the
sse-connector (another PR for the `pekko-connectors` library will follow), so
the user never knows what is happening. The observed behavior appears as if the
stream is still active and just gets stuck at the message that preceeded the
oversized message.
# Solution
This PR introduces a new configuration value:
`pekko.http.sse.oversized-message-handling` which gives the user options for
how the SSE client should handle receiving a message sized over the configured
limit. Four options are available:
1. `fail-stream`: Terminates the stream when oversized messages are
encountered (this is the previous behavior and the new default behavior)
2. `log-and-skip`: Log oversized messages as a warning and continues
processing
3. `truncate`: Truncates and consume oversized messages to fit within limits
and continue
4. `dead-letter`: Sends oversized messages to dead letter queue and continue
# Tests
```
LineParserOversizedSimpleSpec:
LineParser with oversized message handling
- should fail the stream with FailStream strategy (53 milliseconds)
- should skip oversized messages with LogAndSkip strategy (3 milliseconds)
- should truncate oversized messages with Truncate strategy (3 milliseconds)
- should send oversized messages to dead letters with DeadLetter strategy (2
milliseconds)
EventStreamParserOversizedSpec:
An EventStreamParser with oversized message handling
- should parse normal SSE messages correctly with all strategies (6
milliseconds)
- should fail the stream when using FailStream strategy with oversized SSE
line (1 millisecond)
- should skip oversized SSE lines and continue processing with LogAndSkip
strategy (0 milliseconds)
- should truncate oversized SSE lines and continue processing with Truncate
strategy (1 millisecond)
- should send oversized SSE lines to dead letters and continue processing
with DeadLetter strategy (1 millisecond)
- should handle multiple oversized lines in complex SSE events with
LogAndSkip strategy (1 millisecond)
- should handle multiline data with some oversized lines using Truncate
strategy (0 milliseconds)
- should handle streaming SSE data with oversized content across chunks (3
milliseconds)
- should handle event field oversizing with different strategies (7
milliseconds)
- should work with unlimited line sizes when maxLineSize is 0 (2
milliseconds)
LineParserEdgeCasesSpec:
LineParser edge cases
- should handle lines exactly at the size limit (1 millisecond)
- should handle lines one byte over the limit (1 millisecond)
- should handle oversized content spanning multiple chunks (1 millisecond)
- should handle mixed line endings with oversized content (2 milliseconds)
- should handle empty and whitespace lines with size limits (0 milliseconds)
- should handle chunk boundaries at line endings (0 milliseconds)
- should handle consecutive oversized messages (1 millisecond)
- should work with unlimited line size (maxLineSize = 0) (5 milliseconds)
- should handle line ending edge cases (0 milliseconds)
OversizedSseStrategySpec:
OversizedSseStrategy
- should parse valid string values correctly (2 milliseconds)
- should throw IllegalArgumentException for invalid string values (0
milliseconds)
- should convert strategy objects back to strings correctly (1 millisecond)
- should handle case-sensitive strings (0 milliseconds)
- should handle empty and null strings (0 milliseconds)
EventStreamUnmarshallingSimpleSpec:
EventStreamUnmarshalling with oversized message handling
- should fail the stream with FailStream strategy (44 milliseconds)
- should skip oversized content with LogAndSkip strategy (1 millisecond)
- should truncate oversized content with Truncate strategy (1 millisecond)
- should send oversized content to dead letters with DeadLetter strategy (1
millisecond)
```
# Debatable Topics
Calling out specifically some changes which could very reasonably be
otherwise:
## Config values
Size limits of oversized messages are determined by two config values (one
deprecated), `max-event-size` and `max-live-size` which have their own complex
relationships and very small default values. This PR increases the default
values to the smallest reasonable size for modern uses (estimated by looking at
standards applied in browsers. I found none for SSE, but Websocket limits seem
to be within the same spirit and some data on browser limits was available.
The previous limits (4k and 8k) are remarkably small for modern use cases.
As an example: ethereum block chain events (a common enough SSE use case) can
be hundreds of kilobytes. While it is good practice to keep Pekko messages
small, these limits affect what the client is allowed to consume from the
server over HTTP. They aren't primarily about Pekko messages. Larger defaults
seem like a good idea, but it's not core to the primary change proposed in this
PR.
## Unlimited message size
This PR adds the option to disable line and event size limits entirely by
setting those config settings to zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@pekko.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@pekko.apache.org
For additional commands, e-mail: notifications-h...@pekko.apache.org
