hello team. We are now heavily using TEZ 0.10.4.
We have observed that after adding this PR, PipelinedSorter.merger.futures[] may continue to store a large number of elements, and this List<Future> alone may run out of memory by more than 8GB (the maximum number of elements in this ArrayList can be in the tens of millions).This leads to a lot of OOM jobs, and a lot of slow queries. By rolling back TEZ-4542, we've dodged the question for now, and given the short time since the release of tez-0.10.4 (And I think Tez 0.10.4 is in a bad shape.), I would suggest that the community rolls back this PR for the time being, and that this issue may need to be discussed in more detail by all before it can be finalised. Tks! Lisoda
