Hi Scott, For using S3 as artifacts store, you need to make sure that agent machine should have hadoop packages. I would suggest you to install following packages (or its equivalent) on agent machines:
* aws-java-sdk * hadoop-aws Now you should be able to make use of IAM role + aws instance metadata (for storing s3 secrets) to pull those artifacts. Let us know how it goes. On Thu, May 19, 2016 at 9:08 PM, Scott Kinney <[email protected]> wrote: > yes, thanks Shakeel. > I would actually like to have mesos download this from an s3 bucket but > i'm not sure of the path to give the 'uris'. Have you used s3 uris? I'd > like to see an example becasue s3://bucketname/keyname doesn't work nor did > the s3 https endpoint. > I should probably start a new thread for this. > > > ________________________________ > Scott Kinney | DevOps > stem | m 510.282.1299 > 100 Rollins Road, Millbrae, California 94030 > > This e-mail and/or any attachments contain Stem, Inc. confidential and > proprietary information and material for the sole use of the intended > recipient(s). Any review, use or distribution that has not been expressly > authorized by Stem, Inc. is strictly prohibited. If you are not the > intended recipient, please contact the sender and delete all copies. Thank > you. > > ________________________________________ > From: shakeel <[email protected]> > Sent: Thursday, May 19, 2016 7:06 AM > To: [email protected] > Subject: Re: Cannot pull from private docker v1 registry > > Hi Scott, > > I am using a private docker registry on mesos. > > The format is correct .docker/config.json in docker.tar.gz > > Then call the file from the URIs field as > > file:///path/docker.tar.gz > > Hope that helps. > > Kind Regards > Shakeel Suffee > > On 19/05/16 14:27, Scott Kinney wrote: > > I got it. > > > > User error. > > > > Since my docker version is > 1.6 i need to give mesos the > > .docker/config.json in docker.tar.gz. I tried this before but i called > > the archive docker.tgz. That naming convention may not work. > > > > Thanks for your help! > > > > > > > > ------------------------------------------------------------------------ > > Scott Kinney | DevOps > > stem <http://www.stem.com/> | *m* 510.282.1299 > > 100 Rollins Road, Millbrae, California 94030 > > > > This e-mail and/or any attachments contain Stem, Inc. confidential and > > proprietary information and material for the sole use of the intended > > recipient(s). Any review, use or distribution that has not been > > expressly authorized by Stem, Inc. is strictly prohibited. If you are > > not the intended recipient, please contact the sender and delete all > > copies. Thank you. > > ------------------------------------------------------------------------ > > *From:* Joseph Wu <[email protected]> > > *Sent:* Wednesday, May 18, 2016 3:23 PM > > *To:* user > > *Subject:* Re: Cannot pull from private docker v1 registry > > > > The stderr you posted suggests that Mesos successfully fetched your > > .dockercfg. If the following docker pull fails, there should be > > additional logs printed either in the Mesos agent logs, or in the task > > stderr. > > > > Can you check those as well? (And post them here.) > > > > On Wed, May 18, 2016 at 2:29 PM, Scott Kinney <[email protected] > > <mailto:[email protected]>> wrote: > > > > I have a valid .dockercfg credential file on the slave that I pass > > as a uri in the marathon app definition like... > > > > "uris": [ > > "file:///root/.dockercfg" > > > > ], > > > > it fails. > > Mesos sandbox stderr... > > > > I0517 21:45:04.104918 5512 fetcher.cpp:424] Fetcher Info: > > > > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/cf607b5a-b629-46f1-a053-0659b78c4231-S454","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"executable":false,"extract":false,"value":"file:\/\/\/root\/.dockercfg"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/cf607b5a-b629-46f1-a053-0659b78c4231-S454\/frameworks\/cf607b5a-b629-46f1-a053-0659b78c4231-0000\/executors\/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba\/runs\/9c650d01-127c-416b-a00b-5ad09409c76e"} > > I0517 21:45:04.106462 5512 fetcher.cpp:379] Fetching URI > > 'file:///root/.dockercfg' I0517 21:45:04.106475 5512 > > fetcher.cpp:250] Fetching directly into the sandbox directory I0517 > > 21:45:04.106487 5512 fetcher.cpp:187] Fetching URI > > 'file:///root/.dockercfg' I0517 21:45:04.106499 5512 > > fetcher.cpp:167] Copying resource with command:cp '/root/.dockercfg' > > > > '/tmp/mesos/slaves/cf607b5a-b629-46f1-a053-0659b78c4231-S454/frameworks/cf607b5a-b629-46f1-a053-0659b78c4231-0000/executors/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba/runs/9c650d01-127c-416b-a00b-5ad09409c76e/.dockercfg' > > I0517 21:45:04.107993 5512 fetcher.cpp:456] Fetched > > 'file:///root/.dockercfg' to > > > > '/tmp/mesos/slaves/cf607b5a-b629-46f1-a053-0659b78c4231-S454/frameworks/cf607b5a-b629-46f1-a053-0659b78c4231-0000/executors/gridservice.9d35ca3e-1c78-11e6-8664-0242472674ba/runs/9c650d01-127c-416b-a00b-5ad09409c76e/.dockercfg > > > > > > Marathon debug says it can't authenticate. I can pull manually on > > the slave with this credential file. > > Any idea what i'm doing wrong? > > > > > > Scott Kinney | DevOps > > stem | m 510.282.1299 > > 100 Rollins Road, Millbrae, California 94030 > > > > This e-mail and/or any attachments contain Stem, Inc. confidential > > and proprietary information and material for the sole use of the > > intended recipient(s). Any review, use or distribution that has not > > been expressly authorized by Stem, Inc. is strictly prohibited. If > > you are not the intended recipient, please contact the sender and > > delete all copies. Thank you. > > > > > > -- > The information contained in this message is for the intended addressee > only and may contain confidential and/or privileged information. If you are > not the intended addressee, please delete this message and notify the > sender; do not copy or distribute this message or disclose its contents to > anyone. Any views or opinions expressed in this message are those of the > author and do not necessarily represent those of Motortrak Limited or of > any of its associated companies. No reliance may be placed on this message > without written confirmation from an authorised representative of the > company. > > Registered in England 3098391 V.A.T. Registered No. 667463890 > -- Regards, Pradeep Chhetri
