njlaw opened a new issue, #3621: URL: https://github.com/apache/parquet-java/issues/3621
### Describe the bug, including details regarding any error messages, version, and platform. I stumbled on this when an MQTT to parquet program that I'm working on keep getting OOM errors much sooner than I expected. I know the data needs to be retained in memory until it's written to disk, but the JVM was showing significantly more memory growth than the data. The .hprof showed: <img width="917" height="313" alt="Image" src="https://github.com/user-attachments/assets/bb0c9e0d-0b20-4d27-89a8-994bdf8355cf"; /> Tested on 1.17.1 and master@f280d557509da65df7e205dce6898bce9b4a396e It looks like when `endField()` is called, it pushes the current field onto the `previousField` queue instead of replacing the last previous field, so as records are processed, the `previousField` queue grows indefinitely in $O(\text{records} × \text{fields})$ https://github.com/apache/parquet-java/blob/f280d557509da65df7e205dce6898bce9b4a396e/parquet-column/src/main/java/org/apache/parquet/io/ValidatingRecordConsumer.java#L112-L116 ### Component(s) Core -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
