Ryan19929 commented on issue #66982: URL: https://github.com/apache/doris/issues/66982#issuecomment-5581656036
A follow-up finding that affects PR5 (and is worth knowing before PR3/PR4 land): on JDK 17, replacing the full JSON `String` with a streaming writer can *increase* total allocation unless that writer is character-buffered. Gson's `JsonWriter` emits a document as per-token fragments. Writing into an unbuffered `OutputStreamWriter` makes `StreamEncoder` allocate scratch objects on every call — `write(int)` allocates a `char[1]`, `write(String,off,len)` allocates a `char[len]`, and `implWrite` wraps a fresh `HeapCharBuffer`. A 19.6 MB table-metadata payload triggers ~12.7M encoder write calls averaging 1.544 chars each, i.e. roughly 970 MiB of scratch objects per serialization on top of the payload itself. Measured on a length-prefixed writer of the shape PR5 proposes (same fixture, 4 GiB heap, 32 MiB G1 regions, one JVM per configuration, 2 warmups + 5 measured runs, only the test helper changed): | table meta streaming | legacy String path | unbuffered streaming writer | + 8K BufferedWriter | | --- | --- | --- | --- | | off | 1078 MiB/op | 1824 MiB/op | 852 MiB/op | | on | 403 MiB/op | 1153 MiB/op | 173 MiB/op | An 8K `BufferedWriter` drops the encoder write calls from ~12.7M to ~2400. Output bytes are unchanged (buffering only batches the calls into the encoder), peak heap is unaffected (~16 KB of buffer), and the reduction in >= 16 MiB objects that the streaming writer is meant to deliver is preserved. So the "structural removal of the complete JSON String" in the description above is only a net win when the writer is buffered; otherwise it trades one large allocation for a much larger volume of short-lived ones, which on a tight heap also increases premature promotion. I will include the buffering in PR5 itself rather than adding it as a follow-up, and will apply it to every place where Gson writes directly into an `OutputStreamWriter`. The reader side needs no equivalent change: Gson's `JsonReader` already reads through its own 1024 char buffer, so the decoder is not invoked per token. The ~850 MiB that remains with table-meta streaming off comes from the `RuntimeTypeAdapterFactory` `JsonObject`/`LinkedTreeMap` intermediate tree, which is exactly what PR3 removes — the two changes are complementary rather than alternatives. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
