[jira] [Updated] (TEZ-2368) Make the dag number available in Context classes
[ https://issues.apache.org/jira/browse/TEZ-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2368: Attachment: TEZ-2368.3.txt Fixed the typo. Thanks for the review. Committing. Make the dag number available in Context classes Key: TEZ-2368 URL: https://issues.apache.org/jira/browse/TEZ-2368 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-2368.1.txt, TEZ-2368.2.txt, TEZ-2368.3.txt Provide the dag number, which is a unique number, for each dag running within an application in the TezInputContext, TezOutputContext, TezProcessorContext. When containers are re-used, or for external services, this can be used to generate intermediate data to a dag specific directory instead of an application specific directory, where it becomes difficult to differentiate between different dags. The DAG name does provide this - but is not suitable for use in a directory name. Hashing the name is an option, but can lead to collisions. Generating data into a dag specific directory will eventually only be usable when we move away from the default MR handler, or enhance it to support an additional parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2368) Make the dag number available in Context classes
[ https://issues.apache.org/jira/browse/TEZ-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2368: Attachment: TEZ-2368.2.txt Updated patch to rename the method to getDagIdentifier. [~hitesh] - please review. Make the dag number available in Context classes Key: TEZ-2368 URL: https://issues.apache.org/jira/browse/TEZ-2368 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-2368.1.txt, TEZ-2368.2.txt Provide the dag number, which is a unique number, for each dag running within an application in the TezInputContext, TezOutputContext, TezProcessorContext. When containers are re-used, or for external services, this can be used to generate intermediate data to a dag specific directory instead of an application specific directory, where it becomes difficult to differentiate between different dags. The DAG name does provide this - but is not suitable for use in a directory name. Hashing the name is an option, but can lead to collisions. Generating data into a dag specific directory will eventually only be usable when we move away from the default MR handler, or enhance it to support an additional parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2368) Make the dag number available in Context classes
[ https://issues.apache.org/jira/browse/TEZ-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-2368: Attachment: TEZ-2368.1.txt Straightforward patch, with a not so useful unit test. [~hitesh], [~rajesh.balamohan], [~bikassaha] - please review. Make the dag number available in Context classes Key: TEZ-2368 URL: https://issues.apache.org/jira/browse/TEZ-2368 Project: Apache Tez Issue Type: Improvement Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: TEZ-2368.1.txt Provide the dag number, which is a unique number, for each dag running within an application in the TezInputContext, TezOutputContext, TezProcessorContext. When containers are re-used, or for external services, this can be used to generate intermediate data to a dag specific directory instead of an application specific directory, where it becomes difficult to differentiate between different dags. The DAG name does provide this - but is not suitable for use in a directory name. Hashing the name is an option, but can lead to collisions. Generating data into a dag specific directory will eventually only be usable when we move away from the default MR handler, or enhance it to support an additional parameter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
