Kaileshwar16 commented on issue #851: URL: https://github.com/apache/wayang/issues/851#issuecomment-6383120847
@zkaoudi fair point, we cant really guess unique keys from input size alone. Looking at the code again, for the `coGroups` output list we actually dont need to guess at all, since `groups0` and `groups1` are already populated by that line, we can just size it with `groups0.size() + groups1.size()`. For the `HashMap`s, Im thinking we shouldnt try to allocate everything upfront. `CoGroupOperator`'s estimator already assumes a `0.1` reduction factor, and `JavaReduceByOperator` just uses `Collectors.groupingBy` and lets the map grow from default capacity. So we could either start with a modest initial capacity (or `0.1 * input` with a hard cap) and let the maps resize when needed. That might add a tiny bit of resizing overhead when almost every key is unique, so I think it's worth benchmarking low vs high cardinality to find the sweet spot. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
