[
https://issues.apache.org/jira/browse/TEZ-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126372#comment-14126372
]
Bikas Saha commented on TEZ-1539:
---------------------------------
I agree that it may be helpful for user code running inside VMs and IIs to be
able to create a simple case for themselves when they only care about the first
successful task instance's events. It helps in not having to worry about
retries etc.
However, I disagree that this should be the defined approach of the Tez
framework as whole and the internals of the state machines code should be setup
to make this special behavior happen.
1) Events from tasks can come at any time and there may be use cases for
sending and receiving these events without delay. However, for the above
simplification, asap delivery cannot be used since the task may fail after
delivering the first event (in case its supposed to deliver 3). That is why, to
support this simple mode, the events must be sent to the VM/II upon first
successful completion of the task, so that we can guarantee that all events are
delivered only a for a successful task.
2) In addition, there may be use cases where VMs would want to receive events
from the all attempts of a task. I dont think we should restrict that in the
framework internals.
3) For the same DAG, some VMs/IIs may want the simpler behavior and some may
not. So this behavior needs to be local to those VMs/IIs that want it.
The patch is not addressing the above holistically. So I am suggesting the
following alternative.
1) Not make any changes in the core state machines for this simplification.
Events continue to come in and get routed to the desired VMs and IIs. If they
need to be saved for recovery and are not being saved right now, then thats
something that needs to be handled orthogonally under the recovery umbrella
jira. This allows all the above cases to continue to work.
2) IIContext and VMContext should expose a method that allows VMPlugins and
IIPlugins to enable sendEventsOnceOnSuccess mode. This method should ideally be
called during initialization of the VMPlugin/IIPlugin.
3) IIManager and VMManager (the framework wrappers around the plugin code)
implement this behavior. These components are Tez framework code and should be
able to understand and handle attempts/retries etc. When events come to them to
be handed over to the plugins then, today, they simply route them to the
plugin. If the sendEventsOnceOnSuccess mode is enabled then they can buffer
events until the task attempt has completed with success and only then deliver
those events. If the task attempt fails then they can drop the buffered events.
After delivering the events they can drop any further events routed to them.
The above approach enables all the cases described above. In addition, we could
change the return method of the VM/II initialize() method to return a config
object that enables/disables this behavior. That would be incompatible (these
are already marked Unstable) but would ensure that there is no silent behavior.
It will make users think about which mode is needed for their use case and
decide appropriately because they have to return that config object.
> Allow a FIRE_ONCE_ON_SUCCESS model for events generated by user code
> --------------------------------------------------------------------
>
> Key: TEZ-1539
> URL: https://issues.apache.org/jira/browse/TEZ-1539
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-1539.1.wip.txt
>
>
> Specifically for InputInitalizerEvents and VertexManagerEvents.
> Pasting comment from TEZ-1447
> In a majority of cases, events generated by different attempts of the same
> task will be identical - in which case just making use of the event generated
> by the first successful attempt is adequate. Doing something like this manes
> that users don't worry about retries, indices etc - and can just rely on
> receiving a set of events which are to be processed once the vertex succeeds.
> If different attempts of the same workload generate different events -
> processing is likely to be incorrect, since it's very possible for all data
> to be processed (VERTEX successful), then a failure and retry - which
> generates a different event. The initializer doesn't even run at this point,
> since it's already done it's work and is complete. Handling such scenarios,
> likely involves re-running the entire initializer and re-starting the vertex
> which processed the event from scratch. In situations like this, where data
> generated may be different, the best bet is for speculation to be disabled
> (when it's supported), and max-attempts to be set to 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
