[
https://issues.apache.org/jira/browse/TEZ-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286469#comment-14286469
]
Rajesh Balamohan commented on TEZ-1991:
---------------------------------------
We can possibly have the following 2 fields in DME for pipelined data transfer
- last_event_flag (to indicate if it is an incremental event or the last event)
- event_id (to indicate the file number). This could be used to avoid partial
data downloads due to out of order event delivery.
> Introduce additional DataMovementEvent types
> --------------------------------------------
>
> Key: TEZ-1991
> URL: https://issues.apache.org/jira/browse/TEZ-1991
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
>
> IncrementalDataMovementEvents and potentially a FinalDataMovementEvent. This
> is primarily for pipelined data transfer; these serve to reduce the data
> stored in the AM - as well as the events sent to consumers - consolidated
> events / latest event should be adequate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
