[
https://issues.apache.org/jira/browse/TEZ-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120221#comment-14120221
]
Bikas Saha commented on TEZ-1447:
---------------------------------
bq. A registration API with arguments can be added at a later point. For now,
there aren't a lot of events - and just the vertex name should be sufficient.
I am sorry I dont agree with that. It would not be good to have 2 APIs where 1
should be enough. Registering for all events would be register(ENUM.ALL,
Vertices) for anyone who needs to listen to all events. There is no need for
register(vertex) and register(enum, vertex) to be separate.
bq. Parallelism Update is handled differently, since it isn't a state
transition. Since the event is handled differently, it's convenient to provide
th
It should not matter that is not a transition. The publisher just needs to
notify PubSub that ENUM.P_U has occurred for Vertex V. That fact that we have
extended the state machine to notify PubSub about some of the other transitions
is just an implementation detail.
bq. May eventually go with a model where just an Enum is provided.
That is right. I missed that in my last set of comments. The notification could
just be onStateChangeNotify(ENUM, Vertex). There is no need to expose another
set of Event classes/interfaces that need to be casted etc.
bq. I'm not sure what you mean by this. There's a finite set of state changes
that will take place, and the set registered for callbacks is even smaller.
I mean v1 changes state that results in I1 being notified which makes V2 change
state that causes I2 to be notified. From the patch is seems all of this will
happen on the dispatcher thread after it make v1 transition. What I was
suggesting is that PubSub could send an event to the listener entity (via its
vertex), instead of directly invoking the listener callback when the state
changes. This will decouple all these notifications.
bq. Another item which needs to be looked at is whether updates need to be sent
out when a component registers.
Good point. We could follow the java wait-notify convention to check state
before waiting. But that would mean exposing state via a new method. Or PubSub
could immediately notify the listener upon registration because PubSub should
already have the info about that state change in its <Vertex, ENUM> Key to
listener Map.
Another thing would be to add the same notification APIs on the VMs because
they will also need this.
bq. changes based on my previous comment to work with a FIRE_ONCE_ON_SUCCESS
Is this the comment you are referring too?
bq.In a majority of cases, events generated by different attempts of the same
task will be identical - in which case just making use of the event generated
by the first successful attempt is adequate. Doing something like this manes
that users don't worry about retries, indices etc
In that case, that should be probably be done in a separate jira because it
concerns which TezEvents should be routed to VM/II. Looks related to TEZ-1531.
So maybe that one or a new jira. The discussion on that aspect is still open
without conclusion and there is no need to overload this jira with that
discussion. This jira is already non-trivial. Would be good for quick progress
to keep it focused on Vertex State change notifications.
> Handle parallelism updates and versioning w/ custom InputInitializerEvents
> --------------------------------------------------------------------------
>
> Key: TEZ-1447
> URL: https://issues.apache.org/jira/browse/TEZ-1447
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Gunther Hagleitner
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: TEZ-1447.1.wip.txt
>
>
> I'm trying to do dynamic partition pruning through input initializer events
> in Hive. That means that the initializer of a table scan vertex has to
> receive events from all tasks in another vertex (which contain the pruning
> info) before generating tasks to run.
> The problem with the current API I ran into:
> getNumTasks: I'm currently using a busy loop to wait for the num tasks for a
> vertex to be decided (-1 -> x). There's no way around it, because it's the
> only way to find out what number of events to expect (0 is a valid number of
> tasks - so I can't wait for the first to complete).
> With auto-reducer parallelism I have to employ another busy loop. Because I
> might be initially expecting 10 events, which later get's knocked down to 5.
> Since there's no event associated with this, I have to periodically check
> whether I have enough events.
> Versioning: Events have a version number, but I don't know which task they
> are coming from. Thus I can't de-dup events.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
