deniskuzZ opened a new pull request, #540: URL: https://github.com/apache/tez/pull/540
- An announcement carries the source task's output record count so far, not a per-event delta; keep the latest value per input instead of summing events, so a pipelined input's earlier spills are not counted again - Count an input that wrote no rows in the denominator; leaving it out while numInputs keeps it in the multiplier spread the mean of the inputs that had data over the ones that did not - Announce from the composite event path too: which path runs depends only on tez.am.shuffle.auxiliary-service.id, so on the Tez shuffle handler the counter was never updated at all - Multiply before dividing, the mean was truncated first - Saturate the announced count instead of wrapping it through int, and include large records, which bypass OUTPUT_RECORDS -- the VertexManager event already counts both - updateApproximateInputRecords takes the input index - Measured before the fix: 10 inputs, 2 announcing 1000 rows each, read 10,000 instead of 2,000; 4 pipelined events totalling 600 read 450 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
