[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1752: Target Version/s: 0.8.0 (was: 0.7.0) Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.8.0 Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch, TEZ-1752.4.patch, TEZ-1752.5.patch, TEZ-1752.master.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1752: Fix Version/s: (was: 0.7.0) 0.8.0 Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Fix For: 0.8.0 Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch, TEZ-1752.4.patch, TEZ-1752.5.patch, TEZ-1752.master.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1752: -- Attachment: TEZ-1752.master.patch Master: Remove Exception from the abort() method signature ? - Fixed. Also added @unstable MergeManager.close - should this be throwing an InterruptedException - throwing InterruptedException after cleanup Some logs to be removed. - Removed unwanted logs in Fet In PipelinedSorter - isThreadInterrupted - should this cause the loop to exit - Fixed. PipelinedSorter - there's a lot of changes. I think most of them are related to moving the code into a try catch though - All are related to the single tab change flush(). For the cleanup - can we hide this behind a configuration, which defaults to false for now. - Fixed. Added TEZ_RUNTIME_CLEANUP_FILES_ON_INTERRUPT as @unstable with default set to false. For changes related to TEZ-2003, I have created a subtask TEZ-2414. I will post the changes there for branch TEZ-2003. Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch, TEZ-1752.4.patch, TEZ-1752.5.patch, TEZ-1752.master.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1752: -- Attachment: TEZ-1752.4.patch - If the task is running, thread running the processor is interrupted; otherwise taskFuture.get() is interrupted. - Adding the inputs/outputs as and when they are scheduled for initialization in LogicalIOProcessorRuntimeTask Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch, TEZ-1752.4.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1752: -- Attachment: TEZ-1752.5.patch Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch, TEZ-1752.4.patch, TEZ-1752.5.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1752: -- Attachment: TEZ-1752.2.patch In LogicalIOProcessorRuntimeTask - it would be useful to log the interrupt status between each close invocation, and potentially set it if the I/O/P being closed ends up unsetting it. - Done cleanup would behave differently if initialize hasn't been invoked. We may need to track which I O Ps have been initialized - and close just those - In the MergeManager, InterruptedException thrown by MergeThraed.close likely needs to be handled (otherwise it'll end up skipping cleanup?) - Done In the invocation of finalMerge - an IOException is caught, are there specific cases here where this IO exception is actually masking an interrupt ? (and as a result the interrupt status needs to be set) - Done The TezMerger change - should we just change the interface to throw InterruptedException, instead of setting the flag. That's a private method, and will force consumers within the IOs to handle it. - Modified TezMerger to throw InterruptedException UnorderedPartitionedKVWriter / others - in the close method, instead of returning an empty event list - should this just throw an InterruptedException back ? - Done Is the change in the TaskReporter required ? taskFailed shouldn't be invoked after the currentTask has been unregistered. - No, added that since the spurious logs (NPE) were coming up which made it difficult to debug. Master already has the fix for it. Removed the changes in the patch. We likely need to ensure that cleanup / close methods aren't called twice - once during regular cleanup, second during an interrupt while the cleanup is in progress. - Tracking the close() of IPO. This would take care of not making the call twice. Not directly related to interrupts - but an invocation on Task.close() (regular flow) can cause exceptions during Processor close or Input / Output close - which would prevent subsequent Inputs / Outputs from being closed.Do we need to make sure that close() gets invoked on subsequent Inputs / Outputs despite a prior exception ? - Yes, this is needed. Tracking the IPO close() and task.cleanup() in the patch takes care of this. Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1752: -- Attachment: TEZ-1752.3.patch Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1752: - Assignee: Rajesh Balamohan Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1752) Inputs / Outputs in the Runtime library should be interruptable
[ https://issues.apache.org/jira/browse/TEZ-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1752: - Target Version/s: 0.7.0 Inputs / Outputs in the Runtime library should be interruptable --- Key: TEZ-1752 URL: https://issues.apache.org/jira/browse/TEZ-1752 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Rajesh Balamohan Attachments: TEZ-1752.1.patch, TEZ-1752.2.patch, TEZ-1752.3.patch Not possible to preempt tasks without killing containers without this. There's still the problem of Processors not supporting interrupts. We may need API enhancements to either query IPOs on whether they are interrupbtible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
