jerqi commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1593205597
1. ` rss.coordinator.shuffle.nodes.max` is max shuffle nodes which we assign to one shuffle. For example, We have two shuffles, one have 5 partitions, another have 200 partitions, `rss.coordinator.shuffle.nodes.max` is 10. We will assign 5 shuffle nodes to the first shuffle. We will assign another 10 shuffle nodes to the second shuffle. 2. `rss.server.buffer.capacity` is global buffer. We should according to the partition number of shuffle server. We should guarantee that` buffer.capacity` / the number of partition will larger than 2MB, otherwise it will cause too much random io. 3. Every reduce partition is a single buffer. For example: map1 have data reduce 1, reduce2, reduce3. map2 have data reduce 1, reduce2, reduce3. One shuffle node will collect the reduce 1 of map1 and reduce 1 of map2. reduce 1 is a reduce partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
