[
https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237533#comment-16237533
]
Shane Kumpf edited comment on YARN-5428 at 11/3/17 12:20 PM:
-------------------------------------------------------------
Thanks for the feedback [~vinodkv]! After a long hiatus, I've spent a bit of
time looking at the approach you laid out and think it makes sense, despite the
cons.
The approach aligns with the handling of existing credentials and also helps
eliminate the ordering challenges around image localization, since the tokens
will already be available (client config must be localized before docker image
pull).
The major con being that each client will need to add support for a new CLI
option/endpoint so that the user can supply the docker client config via the
client. You mention that most frameworks already have some level of distributed
cache support, however, Credentials and LocalResources aren't handled the same
AFAICT, so a new option needs to be added. I also don't believe it would be
appropriate to add this to GenericOptionsParser, given it only applies to
applications.
Based on the above, my thought is to provide support in distributed shell as
part of this patch and a follow on for support in YARN Native Services.
Enabling this feature for additional clients could follow based on need. Does
that sound reasonable?
was (Author: [email protected]):
Thanks for the feedback @vinodkv! After a long hiatus, I've spent a bit of time
looking at the approach you laid out and think it makes sense, despite the
cons.
The approach aligns with the handling of existing credentials and also helps
eliminate the ordering challenges around image localization, since the tokens
will already be available (client config must be localized before docker image
pull).
The major con being that each client will need to add support for a new CLI
option/endpoint so that the user can supply the docker client config via the
client. You mention that most frameworks already have some level of distributed
cache support, however, Credentials and LocalResources aren't handled the same
AFAICT, so a new option needs to be added. I also don't believe it would be
appropriate to add this to GenericOptionsParser, given it only applies to
applications.
Based on the above, my thought is to provide support in distributed shell as
part of this patch and a follow on for support in YARN Native Services.
Enabling this feature for additional clients could follow based on need. Does
that sound reasonable?
> Allow for specifying the docker client configuration directory
> --------------------------------------------------------------
>
> Key: YARN-5428
> URL: https://issues.apache.org/jira/browse/YARN-5428
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Reporter: Shane Kumpf
> Assignee: Shane Kumpf
> Priority: Major
> Labels: oct16-medium
> Attachments: YARN-5428.001.patch, YARN-5428.002.patch,
> YARN-5428.003.patch, YARN-5428.004.patch,
> YARN-5428Allowforspecifyingthedockerclientconfigurationdirectory.pdf
>
>
> The docker client allows for specifying a configuration directory that
> contains the docker client's configuration. It is common to store "docker
> login" credentials in this config, to avoid the need to docker login on each
> cluster member.
> By default the docker client config is $HOME/.docker/config.json on Linux.
> However, this does not work with the current container executor user
> switching and it may also be desirable to centralize this configuration
> beyond the single user's home directory.
> Note that the command line arg is for the configuration directory NOT the
> configuration file.
> This change will be needed to allow YARN to automatically pull images at
> localization time or within container executor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
