Kaileshwar16 opened a new issue, #851: URL: https://github.com/apache/wayang/issues/851
While looking through JavaCoGroupOperator, I noticed that the grouping maps are initialized based on the estimated number of input records. The output list is also allocated using the combined input sizes. The problem is that these collections hold groups, not individual records. So when a large number of records share the same key, we end up allocating much more memory than needed. For example, if both inputs have 3 million records but only 8 distinct keys, the operator still prepares collections based on millions of entries. I tested the same allocation pattern separately in Java. With a 192 MB heap, the current approach ran into an OutOfMemoryError, while using smaller initial capacities completed successfully. With a 512 MB heap, memory usage was also noticeably lower. This isn't an end-to-end Wayang benchmark yet, but it suggests that the initial allocations could be unnecessarily expensive for low-cardinality inputs. Would it make sense to use more conservative initial capacities here, rather than sizing these collections directly from the input cardinalities? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
