[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated TEZ-1348: - Attachment: tez-1348.txt > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Attachments: tez-1348.patch, tez-1348.patch, tez-1348.txt > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated TEZ-1348: - Attachment: tez-1348.patch > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Attachments: tez-1348.patch, tez-1348.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated TEZ-1348: - Target Version/s: 0.10.0 (was: 0.8.6) > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Attachments: tez-1348.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated TEZ-1348: - Attachment: tez-1348.patch > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Attachments: tez-1348.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1348: - Target Version/s: 0.8.0 (was: 0.6.0) Setup configs required for local mode automatically, instead of relying on changes to tez-site -- Key: TEZ-1348 URL: https://issues.apache.org/jira/browse/TEZ-1348 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Priority: Critical In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in tez-site would work for local mode. Currently the requirement is that tez-site.xml must have fs.defaultFS set to file:///. While that works, it doesn't allow for seamless execution in either local-mode or on a cluster. The main issue here is that when Inputs / Outputs are configured - they use a version of configuration which reads tez-site, and do not use the configuration from the client itself (which is correct behaviour). Not sure what a good way to fix this is 1) It may be possible to override this value each time an instance of Configuration/TezConfiguration is created. One possible way would be to statically add a default resource to Configuration the moment a local client is created. 2) Provide information in the contexts on whether this is local or not. This is fairly ugly, and would get in the way of running mixed mode tasks. Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated TEZ-1348: Summary: Setup configs required for local mode automatically, instead of relying on changes to tez-site (was: Ensure local directories are used when using local-mode) Setup configs required for local mode automatically, instead of relying on changes to tez-site -- Key: TEZ-1348 URL: https://issues.apache.org/jira/browse/TEZ-1348 Project: Apache Tez Issue Type: Sub-task Reporter: Siddharth Seth Priority: Critical In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in tez-site would work for local mode. Currently the requirement is that tez-site.xml must have fs.defaultFS set to file:///. While that works, it doesn't allow for seamless execution in either local-mode or on a cluster. The main issue here is that when Inputs / Outputs are configured - they use a version of configuration which reads tez-site, and do not use the configuration from the client itself (which is correct behaviour). Not sure what a good way to fix this is 1) It may be possible to override this value each time an instance of Configuration/TezConfiguration is created. One possible way would be to statically add a default resource to Configuration the moment a local client is created. 2) Provide information in the contexts on whether this is local or not. This is fairly ugly, and would get in the way of running mixed mode tasks. Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
