[
https://issues.apache.org/jira/browse/TEZ-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131070#comment-14131070
]
Bikas Saha commented on TEZ-1539:
---------------------------------
bq.Agreed. The last patch done by Sid should be saving those events into
recovery log within the source vertex itself.
[~hitesh] from the patch it looks like II events are being stored in the
destination vertex and not the source vertex, unless I am reading it wrong
{code}
if (!isEventFromVertex(vertex, tezEvent.getSourceInfo())) {
- continue;
+ if
(tezEvent.getEventType().equals(EventType.ROOT_INPUT_INITIALIZER_EVENT)) {
+ recoveryEvents.add(tezEvent);
+ } else {
+ continue;
+ }
}{code}
It might still work but shouldn't the flow be consistent with all other events
where the events are stored in their source vertex? Might break things down the
road.
> Allow a FIRE_ONCE_ON_SUCCESS model for events generated by user code
> --------------------------------------------------------------------
>
> Key: TEZ-1539
> URL: https://issues.apache.org/jira/browse/TEZ-1539
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-1539.1.wip.txt, TEZ-1539.2.txt, TEZ-1539.3.txt,
> TEZ-1539.4.txt
>
>
> Specifically for InputInitalizerEvents and VertexManagerEvents.
> Pasting comment from TEZ-1447
> In a majority of cases, events generated by different attempts of the same
> task will be identical - in which case just making use of the event generated
> by the first successful attempt is adequate. Doing something like this manes
> that users don't worry about retries, indices etc - and can just rely on
> receiving a set of events which are to be processed once the vertex succeeds.
> If different attempts of the same workload generate different events -
> processing is likely to be incorrect, since it's very possible for all data
> to be processed (VERTEX successful), then a failure and retry - which
> generates a different event. The initializer doesn't even run at this point,
> since it's already done it's work and is complete. Handling such scenarios,
> likely involves re-running the entire initializer and re-starting the vertex
> which processed the event from scratch. In situations like this, where data
> generated may be different, the best bet is for speculation to be disabled
> (when it's supported), and max-attempts to be set to 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
