That is incredibly useful info! Thanks :)
-- Aaron Carey Production Engineer - Cloud Pipeline Industrial Light & Magic London 020 3751 9150 ________________________________ From: Pradeep Chhetri [[email protected]] Sent: 19 May 2016 18:17 To: [email protected]; [email protected] Subject: Re: Cannot pull from private docker v1 registry Hi Scott, For using S3 as artifacts store, you need to make sure that agent machine should have hadoop packages. I would suggest you to install following packages (or its equivalent) on agent machines: * aws-java-sdk * hadoop-aws Now you should be able to make use of IAM role + aws instance metadata (for storing s3 secrets) to pull those artifacts. Let us know how it goes. On Thu, May 19, 2016 at 9:08 PM, Scott Kinney <[email protected]<mailto:[email protected]>> wrote: yes, thanks Shakeel. I would actually like to have mesos download this from an s3 bucket but i'm not sure of the path to give the 'uris'. Have you used s3 uris? I'd like to see an example becasue s3://bucketname/keyname doesn't work nor did the s3 https endpoint. I should probably start a new thread for this. ________________________________ Scott Kinney | DevOps stem | m 510.282.1299 100 Rollins Road, Millbrae, California 94030 This e-mail and/or any attachments contain Stem, Inc. confidential and proprietary information and material for the sole use of the intended recipient(s). Any review, use or distribution that has not been expressly authorized by Stem, Inc. is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. Thank you. ________________________________________ From: shakeel <[email protected]<mailto:[email protected]>> Sent: Thursday, May 19, 2016 7:06 AM To: [email protected]<mailto:[email protected]> Subject: Re: Cannot pull from private docker v1 registry Hi Scott, I am using a private docker registry on mesos. The format is correct .docker/config.json in docker.tar.gz Then call the file from the URIs field as file:///path/docker.tar.gz Hope that helps. Kind Regards Shakeel Suffee On 19/05/16 14:27, Scott Kinney wrote: > I got it. > > User error. > > Since my docker version is > 1.6 i need to give mesos the > .docker/config.json in docker.tar.gz. I tried this before but i called > the archive docker.tgz. That naming convention may not work. > > Thanks for your help! > > > > ------------------------------------------------------------------------ > Scott Kinney | DevOps > stem <http://www.stem.com/> | *m* 510.282.1299 > 100 Rollins Road, Millbrae, California 94030 > > This e-mail and/or any attachments contain Stem, Inc. confidential and > proprietary information and material for the sole use of the intended > recipient(s). Any review, use or distribution that has not been > expressly authorized by Stem, Inc. is strictly prohibited. If you are > not the intended recipient, please contact the sender and delete all > copies. Thank you. > ------------------------------------------------------------------------ > *From:* Joseph Wu <[email protected]<mailto:[email protected]>> > *Sent:* Wednesday, May 18, 2016 3:23 PM > *To:* user > *Subject:* Re: Cannot pull from private docker v1 registry > > The stderr you posted suggests that Mesos successfully fetched your > .dockercfg. If the following docker pull fails, there should be > additional logs printed either in the Mesos agent logs, or in the task > stderr. > > Can you check those as well? (And post them here.) > > On Wed, May 18, 2016 at 2:29 PM, Scott Kinney > <[email protected]<mailto:[email protected]> > <mailto:[email protected]<mailto:[email protected]>>> wrote: > > I have a valid .dockercfg credential file on the slave that I pass > as a uri in the marathon app definition like... > > "uris": [ > "file:///root/.dockercfg" > > ], > > it fails. > Mesos sandbox stderr... > > I0517 21:45:04.104918 5512 fetcher.cpp:424] Fetcher Info: > > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/cf607b5a-b629-46f1-a053-0659b78c4231-S454","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":false,"value":"file:\/\/\/root\/.dockercfg"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/cf607b5a-b629-46f1-a053-0659b78c4231-S454\/frameworks\/cf607b5a-b629-46f1-a053-0659b78c4231-0000\/executors\/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba\/runs\/9c650d01-127c-416b-a00b-5ad09409c76e"} > I0517 21:45:04.106462 5512 fetcher.cpp:379] Fetching URI > 'file:///root/.dockercfg' I0517 21:45:04.106475 5512 > fetcher.cpp:250] Fetching directly into the sandbox directory I0517 > 21:45:04.106487 5512 fetcher.cpp:187] Fetching URI > 'file:///root/.dockercfg' I0517 21:45:04.106499 5512 > fetcher.cpp:167] Copying resource with command:cp '/root/.dockercfg' > > '/tmp/mesos/slaves/cf607b5a-b629-46f1-a053-0659b78c4231-S454/frameworks/cf607b5a-b629-46f1-a053-0659b78c4231-0000/executors/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba/runs/9c650d01-127c-416b-a00b-5ad09409c76e/.dockercfg' > I0517 21:45:04.107993 5512 fetcher.cpp:456] Fetched > 'file:///root/.dockercfg' to > > '/tmp/mesos/slaves/cf607b5a-b629-46f1-a053-0659b78c4231-S454/frameworks/cf607b5a-b629-46f1-a053-0659b78c4231-0000/executors/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba/runs/9c650d01-127c-416b-a00b-5ad09409c76e/.dockercfg > > > Marathon debug says it can't authenticate. I can pull manually on > the slave with this credential file. > Any idea what i'm doing wrong? > > > Scott Kinney | DevOps > stem | m 510.282.1299 > 100 Rollins Road, Millbrae, California 94030 > > This e-mail and/or any attachments contain Stem, Inc. confidential > and proprietary information and material for the sole use of the > intended recipient(s). Any review, use or distribution that has not > been expressly authorized by Stem, Inc. is strictly prohibited. If > you are not the intended recipient, please contact the sender and > delete all copies. Thank you. > > -- The information contained in this message is for the intended addressee only and may contain confidential and/or privileged information. If you are not the intended addressee, please delete this message and notify the sender; do not copy or distribute this message or disclose its contents to anyone. Any views or opinions expressed in this message are those of the author and do not necessarily represent those of Motortrak Limited or of any of its associated companies. No reliance may be placed on this message without written confirmation from an authorised representative of the company. Registered in England 3098391 V.A.T. Registered No. 667463890 -- Regards, Pradeep Chhetri
