[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536352#comment-15536352 ]
Kuhu Shukla commented on TEZ-3362: ---------------------------------- [~hitesh], we can move the generic service plugin effort, which would provide APIs for shuffle Port and scheduler service discovery; allowing path based deletion to a separate JIRA under TEZ-3334. The DagAppMaster will initialize this "Deletion Service" in a manner similar to how TaskSchedulers and ContainerLaunchers are initialized today. Does that sound as a good starting point? Are we ok to have this patch in for an initial POC in a deployed environment? Let me know and I can open a JIRA for the proposal. Thanks a lot and looking forward to more comments and ideas! > Delete intermediate data at DAG level for Shuffle Handler > --------------------------------------------------------- > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task > Reporter: Jonathan Eagles > Assignee: Kuhu Shukla > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch, TEZ-3362.004.patch, TEZ-3362.005.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)