shameersss1 commented on pull request #60:
URL: https://github.com/apache/tez/pull/60#issuecomment-1032261958
> @shameersss1: I'm more than interested in this patch, let me have some
time to review it this needs more thorough testing than TEZ-4129 as TEZ-4129
was on the unhappy code path (failed attempts), but this one seriously affects
shuffle could you please describe what kind of testing process have you done
with this patch?
@abstractdog - Thanks for showing interest to review. It has been pending
for a while now.
The high level idea behind this feature is that, Whenever all the dependent
vertex of a particular vertex have succeeded we delete the vertex shuffle data
of that particular/parent vertex.
Testing Procedure
1. I picked a query which spawns a big dag (preferably some TPC-DS query)
which runs to quite some time. I changed number of max reducers to 1 so that
the final stage takes time
2. I checked if the shuffle data of the parent vertex are deleted when all
the dependent vertex succeeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
