Croway commented on PR #30368:
URL: https://github.com/apache/camel/pull/30368#issuecomment-7006013278

   I ran three checks against a real Redis (test-infra container) on 
64454ce1dd, with the merge base 45da7713c0 as the control.
   
   **1. The default filter also limits the size of a stored exchange, which 
Redis users did not have before.** With no `deserializationFilter` and no 
`jdk.serialFilter`, `unmarshallExchange` 
(`RedisAggregationRepository.java:685`) falls back to 
`DeserializationFilterHelper.DEFAULT_DESERIALIZATION_FILTER`. Besides the class 
allow-list, that pattern sets `maxdepth=20;maxrefs=10000;maxbytes=10485760`. 
When a stored exchange goes over a limit, `get()` throws for that correlation 
key on every following message, and the aggregation never completes:
   
   | route | PR head | merge base |
   |---|---|---|
   | a strategy that appends to a `List` body, 15 messages × 1000 strings, 
`completionSize(15)` | messages 11 to 15 fail with `Cannot deserialize the 
exchange of true in the Redis hash grouped` ← `InvalidClassException: filter 
status: REJECTED`, and nothing completes | completes |
   | `AggregationStrategies.string()`, an 11 MB body then a second message, 
`completionSize(2)` | the 2nd message fails the same way, and nothing completes 
| completes |
   
   JDBC, Cassandra and LevelDB have had the same limits since 4.21, so the 
default is consistent with them. But the new upgrade-guide entry and 
`redis.adoc` only describe the class allow-list, so anyone who aggregates large 
batches in Redis today will hit this after the upgrade. Could the entry list 
the three limits and show how to raise them (a custom pattern without them, or 
with larger values)?
   
   **2. One unreadable completed exchange stops recovery for all the others.** 
`recover()` (`:266`) now throws when the filter rejects an entry. 
`AggregateProcessor.RecoverTask` catches only around the whole scan loop, so 
every run stops at that entry. I stored 10 completed exchanges, one of them 
with 12000 strings in the body, and started an aggregator with 
`recoveryInterval=300`. After 5 s (about 16 runs), the PR head had recovered 0 
of 10. The rejected entry came first in `HKEYS` order, and each run logs `Error 
during recover task ... Cannot deserialize the exchange of ...-0004 in the 
Redis hash rec-completed`. The merge base recovered 10 of 10. The JDBC 
repository also throws from `recover()`, so a per-exchange catch in 
`RecoverTask` could be a follow-up. Still, this PR adds new ways for Redis to 
throw there; logging a WARN in `recover()` and skipping the entry would avoid 
it.
   
   **3. The startup check downloads the whole repository on every start.** 
`warnUnreadableEntries` (`:574`) runs on every `doStart`, not only the first 
one after the upgrade, and `map.entrySet(SCAN_COUNT)` HSCANs the values too. 
With 100 completed exchanges of 1 MB, Redis' `total_net_output_bytes` grew by 
100 MB on each `start()` (459 ms and 206 ms), against 0 MB (36 ms) on the merge 
base. Stopping after the first `SCAN_COUNT` entries would still spot a hash 
written by an older version.
   
   _Claude Code on behalf of Croway_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to