[
https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-1752:
----------------------------------
Attachment: TEZ-1752.2.patch
In LogicalIOProcessorRuntimeTask - it would be useful to log the interrupt
status between each close invocation, and potentially set it if the I/O/P being
closed ends up unsetting it.
- Done
cleanup would behave differently if initialize hasn't been invoked. We may need
to track which I O Ps have been initialized - and close just those - In the
MergeManager, InterruptedException thrown by MergeThraed.close likely needs to
be handled (otherwise it'll end up skipping cleanup?)
- Done
In the invocation of finalMerge - an IOException is caught, are there specific
cases here where this IO exception is actually masking an interrupt ? (and as a
result the interrupt status needs to be set)
- Done
The TezMerger change - should we just change the interface to throw
InterruptedException, instead of setting the flag. That's a private method, and
will force consumers within the IOs to handle it.
- Modified TezMerger to throw InterruptedException
UnorderedPartitionedKVWriter / others - in the close method, instead of
returning an empty event list - should this just throw an InterruptedException
back ?
- Done
Is the change in the TaskReporter required ? taskFailed shouldn't be invoked
after the currentTask has been unregistered.
- No, added that since the spurious logs (NPE) were coming up which made it
difficult to debug. Master already has the fix for it. Removed the changes in
the patch.
We likely need to ensure that cleanup / close methods aren't called twice -
once during regular cleanup, second during an interrupt while the cleanup is in
progress.
- Tracking the close() of IPO. This would take care of not making the call
twice.
Not directly related to interrupts - but an invocation on Task.close() (regular
flow) can cause exceptions during Processor close or Input / Output close -
which would prevent subsequent Inputs / Outputs from being closed.Do we need to
make sure that close() gets invoked on subsequent Inputs / Outputs despite a
prior exception ?
- Yes, this is needed. Tracking the IPO close() and task.cleanup() in the patch
takes care of this.
> Inputs / Outputs in the Runtime library should be interruptable
> ---------------------------------------------------------------
>
> Key: TEZ-1752
> URL: https://issues.apache.org/jira/browse/TEZ-1752
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch
>
>
> Not possible to preempt tasks without killing containers without this.
> There's still the problem of Processors not supporting interrupts. We may
> need API enhancements to either query IPOs on whether they are interrupbtible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
