[
https://issues.apache.org/jira/browse/TEZ-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709710#comment-15709710
]
Jonathan Eagles commented on TEZ-3222:
--------------------------------------
Thanks for the feedback [~mingma]. I share many of the concerns you have on
this issue. Many of the issue have ongoing efforts in other JIRAs that will
help to improve the code as you have suggested. Let me know any concern have
based on my inline comments.
bq. This includes EdgeManagerPluginOnDemand API, Is there any 3rd party plugin
that depends on the old signature?
Neither pig nor hive nor cascading depend on this signature change. It is my
understanding that we agreed to this known API change.
bq. CompositeRoutedDataMovementEvent has comment Event used by user code to
send information between tasks.. How is that used by user node? It seems the
relevant part is EdgeManagerPlugin provider, which needs to return
CompositeEventRouteMetadata.
Copied this description verbatim from DataMovementEvent. Do you have a
suggestion for better wording here? I agree it is not very helpful as is.
bq. Should routeInputSourceTaskFailedEventToDestination be optimized as well?
I think this is possible. Can optimizing source task failed be a separate
follow-on JIRA? This one is getting difficult to maintain and I won't have
cycles immediately to address this.
bq. Given CompositeRoutedDataMovementEvent adds target indices compared to
CompositeRoutedDataMovementEvent, will it help if using inheritance?
An earlier thread addressed this offline to allow these events to evolve
separately since the vision is to allow the details to hidden as the number of
compound types increases.
bq. Maybe the following block in ShuffleInputEventHandlerImpl and
ShuffleInputEventHandlerOrderedGrouped can be moved to a common function used
by all cases?
I have the same concern as you in this case. There is a separate effort to
combine the extremely similar code in these two modules. I think it will be
better to address this in that JIRA.
> Reduce messaging overhead for auto-reduce parallelism case
> ----------------------------------------------------------
>
> Key: TEZ-3222
> URL: https://issues.apache.org/jira/browse/TEZ-3222
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3222.1.patch, TEZ-3222.2.patch, TEZ-3222.3.patch,
> TEZ-3222.4.patch, TEZ-3222.5.patch, TEZ-3222.6.patch, TEZ-3222.7.patch
>
>
> A dag with 15k x 1000k vertex may auto-reduce to 15k x 1. And while the data
> size is appropriate for 1 task attempt, this results in an increase in task
> attempt message processing of 1000x.
> This jira aims to reduce the message processing in the auto-reduced task
> while keeping the amount of message processing in the AM the same or less.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
