[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114352#comment-16114352 ] Hadoop QA commented on YARN-5621: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} YARN-5621 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5621 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827955/YARN-5621.5.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16709/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Jian He >Assignee: Jian He > Labels: oct16-hard > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114150#comment-16114150 ] Yang Wang commented on YARN-5621: - {code:title=LinuxContainerExecutor.java} protected void createSymlinkAsUser(String user, File privateScriptFile, String userScriptFile) throws PrivilegedOperationException { String runAsUser = getRunAsUser(user); ... ... {code} I think we should use containerUser instead of runAsUser here. Because it may cause "Invalid command" in container-executor when getRunAsUser return nonsecureLocalUser. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Jian He >Assignee: Jian He > Labels: oct16-hard > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524603#comment-15524603 ] Chris Douglas commented on YARN-5621: - That summary of work seems about right, thanks for putting it together. You raise excellent points about error handling. Your sketch includes a channel communicating which resources were (un)successfully linked. The script-driven approach handles this in v05 by writing a separate bash script and invoking the CE for each symlink (which, to be fair, isn't exactly "lightweight" when compared to extending {{ContainerLocalizer}}). In v05, a failure affects only one resource, but to take your earlier example linking a batch of resources in the script: how would one handle partial failures? What's the state of the container and resources when the script invocation fails? On the CL proposal: either the CI initiates the symlink request to the {{ResourceLocalizationService}} after download, or the two operations are contained within that service. The complexity is comparable. The 2-phase protocol you sketch (CI initiates download, then link) adds a gap when the CL could be shut down before it receives the {{LINK}} commands (causing two CL launches), but even a short timeout would likely cover that. A single-message annotating the resource (download+symlink) could add states to {{LocalizedResource}} if it were to notify starting containers directly (current code) or handoff to the RLS for symlink. In this case, the protocol to the {{ContainerImpl}} is simpler (resending/retry is idempotent b/c it doesn't care if the download or symlink failed). Both {{FetchSuccessTransition}} and {{LocalizedResourceTransition}} would need to send {{LocalizerResourceRequestEvent}} for running containers to symlink. A failed symlink would look like a failed download to the CI. Start container is unaffected. For the CL itself... sure, {{ResourceLocalizationSpec}} needs an another field for symlinks. This side is pretty straightforward, right? > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522339#comment-15522339 ] Jian He commented on YARN-5621: --- I still have hesitation on this approach to spawn an additional process to only create symlinks, as it is heavier than directly invoking container-executor and implementation is more complicated. I checked the code, it seems not that simple to make the whole flow work. Concretely , - When container creates the symlink, containerImpl needs to send a new LINK event to ResourceLocalizationService which will create a new LocalizerRunner thread- a number of refactors need to be made here so that all parameters are made available to the LocalizerRunner. - Then LocalizerRunner#findNextResource internally the logic need to be forked to send over symlinks instead of resources. - LocalizerHeatBeatRequest: One LINK object with symlink payloads need to be added in the heartBeat request, - LocalizerHeatBeatResponse: Two more objects LINK_FAILED and LINK_SUCCEED need to be added, and also the resource the link corresponds to. - Once done, ResourceLocalizationService need to send LINK_FAILED and LINK_SUCCEED events back to ContainerImpl - ContainerImpl state-machine gets complicated mainly because the symlink operation becomes asynchronous. -- first it needs to keep track of all original LINK requests so that when ResourceLocalizationService sends the results back, it knows which resource this request was originally correspond to. User needs to query the status. -- new state-transitions also need to be added to handle the new LINK_FAILED and LINK_SUCCEEDED events. I also would like to know what changes I need to make if we choose the container-executor approach. [~vvasudev], what's your opinion ? > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510866#comment-15510866 ] Chris Douglas commented on YARN-5621: - bq. I think I understand your approach now, basically, [...] Yes, that's the gist of it. The {{ContainerLocalizer}} manages the private cache as the user, with that user's cluster credentials. Running containers start a CL to download private resources and/or create symlinks. bq. Is it starting both instances now? Not sure if I read the code wrong... It seems not the case. Based on the code, if it's an already existing resource, it will NOT start the ContainerLocalizer. [container start] For different applications? It should. For container start, I don't remember offhand if the {{ContainerLocalizer}} spawn is delayed until at least one dependent resource is not claimed, but IIRC it starts if at least one resource is not downloaded. Either way, CLs could start in race for a resource _R_, and only one would (successfully) download it. Resources aren't claimed when the CL launches, only when it heartbeats in. [CL proposal] For running container localization and for rollback, the CL will download the resource (again) and/or create the symlink to the running container. If multiple containers/applications request the same resource, it doesn't matter if it's a mix of new/running containers requesting a resource _R_. Only running/rollback containers will send symlink commands to their CL. bq. This approach may not be easily worked for the new containers without structural change, when localizer is started, the work-dirs are not setup yet. Again, container start is unaffected; new containers will not send {{LINK}} commands to the CL. Only _running_ containers will start a CL that receives {{LINK}} commands, after the work dirs are created and the container has started. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508689#comment-15508689 ] Jian He commented on YARN-5621: --- Thanks for your explanation, some parts I'm missing: bq. Both will start ContainerLocalizer instances Is it starting both instances now? Not sure if I read the code wrong... It seems not the case. Based on the code, if it's an already existing resource, it will NOT start the ContainerLocalizer. e.g. below comments in ResourceLocalizationService. Could you point me the right cod... maybe I'm missing something. {code} /* * Multiple containers will try to download the same resource. So the * resource download should start only if * 1) We can acquire a non blocking semaphore lock on resource * 2) Resource is still in DOWNLOADING state */ {code} bq. it may be used by both, concurrently. This approach may not be easily worked for the new containers without structural change, because for new containers, when localizer is started, the work-dirs are not setup yet. It cannot create symlinks on localization for new containers. The work-dirs are created later when launching the container. bq. for services that upgrade over minutes/hours, Not only for upgrades, it's also used by Tez for localizing resources on demand. I think I understand your approach now, basically, To create the symlinks, 1. we start the localizer process, 2. send the symlinks over on localizer heartbeat, 3. localizer process create symlinks. One question is, should we keep the localizer process around or terminates it immediately after symlinks created. If we keep it around for certain time, we need to add some life timeout for the localizer process. If it's immediately terminated, that means every localize request by AM would spawn/kill a new process, and I think that could churn over the machine resources. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15507320#comment-15507320 ] Chris Douglas commented on YARN-5621: - I think I see where the CL proposal was unclear. It is an alternative to CE changes; container start remains as-is. The proposal was scoped only to localizing resources for running containers. The CE is agnostic to new/running containers for an application- it may be used by both, concurrently. By adding a new command {{LINK}} to its protocol, the NM can instruct the {{ContainerLocalizer}} to create a symlink to a resource for a running container. Again, these commands could be grouped. {quote} > a case that already exists for containers on the same node requesting the > same resource Do you mean this is an existing implemented functionality or this is an existing use-case? {quote} Neither. The case where running containers (c ~1x~, c ~2y~) for different applications (a ~1~, a ~2~) request the same resource _R_ exists. Both will start {{ContainerLocalizer}} instances, but only one will download the resource to the private cache. In the CL proposal, this is the same as rollback, where the CL starts, heartbeats, then receives a command to LINK an existing resource without downloading anything. By "a case that already exists", I meant it's a case the CL proposal handles implicitly. bq. yeah, I feel it's inefficient to start a localizer process to only create symlinks.. No question. But if localizing a new resource takes a few seconds, for services that upgrade over minutes/hours, then a few hundred milliseconds is not worth adding {{RUN_SCRIPT}} to the CE. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505601#comment-15505601 ] Jian He commented on YARN-5621: --- My original thought is to create the symlinks right after localization completes. Then I realized for existing resource, no localizer process is created, and, yeah, I feel it's inefficient to start a localizer process to only create symlinks.. Not sure I understand below right.. bq. a case that already exists for containers on the same node requesting the same resource Do you mean this is an existing implemented functionality or this is an existing use-case? I think the former is not true. The latter is true which is currently done by the container_launch script to create symlinks > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504180#comment-15504180 ] Chris Douglas commented on YARN-5621: - bq. this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Sorry, I'm missing something. If the {{ContainerLocalizer}} supports a command to create symlinks to localized resources- a case that already exists for containers on the same node requesting the same resource- then how is that case distinguished from rollback? The container does need to start a {{ContainerLocalizer}} just to write some symlinks for the running container, which is inefficient. On the other hand, all symlinks for all containers from an application could be updated in the same invocation. When you say it does not work, are you noting the inefficiency of this flow, or is there a correctness problem? > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15502377#comment-15502377 ] Jian He commented on YARN-5621: --- Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the current approach is fine > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491452#comment-15491452 ] Chris Douglas commented on YARN-5621: - bq. This may be a viable approach, we need to change the localizer heartbeat to send the symlink path. The heartbeat already carries a payload with commands to the localizer. Including actions to symlink resources already fetched isn't that dire a change to either the ContainerLocalizer or the resource state machine, is it? The transition needs to send a LINK request to all localizers that were waiting in case the download failed. bq. But if we want to create all symlinks in one go, this approach will not work. This isn't going to be a transaction on the FS regardless, but can you explain this requirement? If symlink-on-download is disqualifying, then the container could still coordinate grouped symlinks by grouping LINK requests to a localizer. It rearranges the event flows awkwardly, but it's supportable... > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482929#comment-15482929 ] Hadoop QA commented on YARN-5621: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s {color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 15 unchanged - 2 fixed = 15 total (was 17) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 32s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 11s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827955/YARN-5621.5.patch | | JIRA Issue | YARN-5621 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 63f480d6d115 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cc01ed70 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13077/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13077/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apa
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15480072#comment-15480072 ] Hadoop QA commented on YARN-5621: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 53s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 18s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 267 unchanged - 0 fixed = 269 total (was 267) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 50s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 34s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827893/YARN-5621.4.patch | | JIRA Issue | YARN-5621 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux bd41fb39a6b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / bee9f57 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/13071/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | cc | https://builds.apache.org/job/PreCommit-YARN-Build/13071/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/13071/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13071/arti
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15479168#comment-15479168 ] Allen Wittenauer commented on YARN-5621: I'll send you a note in private. ;) > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15478543#comment-15478543 ] Chris Douglas commented on YARN-5621: - bq. FWIW, I'd love to see us drop the container launch script. I haven't tried it, but I suspect we can do lots of fun things with the env vars. For all containers, we have (1) NM constants (2) some user args we verify (e.g., container ID matches the token, is correctly formatted, etc.) used as args to the CE (which should validate that each of these args conforms to a schema). These are the args used to build paths. All other args (3) the user can control should be written to the container launch script, which is executed with the same permissions the container would have. The intent was to have all quoting games happen after we've switched to the user's context, and after we've discarded the NM environment. The implementation may have gaps, but is there a problem with the concept? This JIRA follows a similar pattern, but without validation of args in the CE. If it were restricted s.t. the source had a fixed format in {{nmPrivate}} and the destination was derived from a formatted {{ContainerID}}, it could have comparable guarantees as the container start. Unless the resource is public, could this avoid modifying the CE by moving the symlink to the {{ContainerLocalizer}}? It could receive a symlink command on a heartbeat, it's already running as the user, it may already be running to download the resource... > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477967#comment-15477967 ] Allen Wittenauer commented on YARN-5621: Actually, that reminds me where *is* the input validation here? Every time we use bash, we're opening the door to all sorts of fun metacharacter issues. Is it possible for a user to try to get a symlink for: {code} 1\\;rm -rf / 1"\\;rm -rf / 1\\\;;rm -rf / 1\\;;*;rm -rf / {code} ... etc. FWIW, I'd love to see us drop the container launch script. I haven't tried it, but I suspect we can do lots of fun things with the env vars. Especially if it is using execlp instead of execl. Just because we wrap stuff in quotes doesn't mean that code is magically safe. (and because set -e, pipefail, etc aren't set in that launch script, it just makes it a bigger/easier target.) > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477673#comment-15477673 ] Allen Wittenauer commented on YARN-5621: Yup. Multiple invocations are easily worked around anyway by creating a directory structure mapping, similar to what OS package managers have been doing for decades. Doing it that way also allows for input validation, which is completely missing from any script-based method. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477639#comment-15477639 ] Chris Douglas commented on YARN-5621: - bq. Because the passed in symlink path is an absolute path Yes, obviously. :) I'm asking why this is an absolute path, if (per the design doc) the symlink is still relative to the container's working dir. bq. later on we need to create multiple symlinks in a single operation as done in current container_launch script, because if there is a large number of local Resources to be localized, we don't want to invoke the binary for each of them. Invoking the binary for each resource isn't so dire. Linking a group of resources only if they're all successfully localized could be useful for services/upgrades, though. bq. I guess the question is why the original container_launch script is not done in this way? I think Allen's point is that the TC/CE binaries have avoided abstraction and other conventional good taste to reduce the attack surface. If the CE can only run scripts that were written by the NM to a specific, restricted directory, it can only run them as the user in a destination following the NM schema, etc. that makes it harder to involve the CE in an attack. If the CE can invoke one stage without preconditions guaranteed by the previous stage, as {{--run-script}} may allow, that's substantively different from the existing behavior. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475973#comment-15475973 ] Jian He commented on YARN-5621: --- bq. the container launch script is pretty self contained, is mostly controlled by the NM, and there are other actions in the pipeline. Running a generic script without any of that extra baggage around it seems to be greatly expanding the footprint of c-e. I still don't see the difference. Both are controlled by NM. The launch_container.sh runs any arbitrary user supplied command and this feature does the same thing. If you just pass dummy objects to other parameters of container launch command, it's essentially doing exactly the same thing. bq. Why can't the directory structure be built by the NM and it just be a chown operation by c-e? I guess the question is why the original container_launch script is not done in this way? Even if it's done this way, should we then add a new 'chown' API in the container-executor ? or should it belong to the API of creating symlinks (similar as a script)? It's much easier to follow existing container_launch code which takes care of all environments, rather than inventing new. Also, later on we need to create multiple symlinks in a single operation as done in current container_launch script, because if there is a large number of local Resources to be localized, we don't want to invoke the binary for each of them. > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org