Hi Sam, Glad that it works. I'll look into the issue further to see what actually is causing the issue.
Terence On Tue, Feb 23, 2016 at 11:47 AM, Sam William <[email protected]> wrote: > HI Terence, > Yes, this hack should be OK for the time-being. My user definitely > available on the cluster and I have been running map reduce jobs regularly. > > Sam > > > On Feb 22, 2016, at 3:06 PM, Terence Yim <[email protected]> wrote: > > > > Is your user available in the cluster? In a non-secure cluster, you can > > impersonate a user when creating the FileSystem object. > > > > FileSystem fs = UserGroupInformation.createRemoteUser("some_user").doAs( > > ... > > ) > > > > > > On Mon, Feb 22, 2016 at 2:39 PM, Sam William <[email protected]> > wrote: > > > >> Yeah, current user is ‘yarn’, and not the user that launched the job. > >> > >> 14:35:25.400 [TwillContainerService] INFO > >> o.a.twill.example.yarn.HelloWorld - Current User: yarn (auth:SIMPLE) > >> > >>> On Feb 22, 2016, at 1:52 PM, Terence Yim <[email protected]> wrote: > >>> > >>> Hi Sam, > >>> > >>> I see. In that case, try to create the Filesystem object with a > >>> UserGroupInformation. First, you might want to verify you have the > right > >>> user in the UGI: > >>> > >>> LOG.info("Current User: {}", UserGroupInformation.getCurrentUser()); > >>> FileSystem fs = UserGroupInformation.getCurrentUser().doAs(new > >>> PrivilegedExceptionAction<FileSystem>() { > >>> @Override > >>> public FileSystem run() throws Exception { > >>> return FileSystem.get(new Configuration()); > >>> } > >>> }); > >>> > >>> Terence > >>> > >>> > >>> On Mon, Feb 22, 2016 at 1:33 PM, Sam William <[email protected]> > >> wrote: > >>> > >>>> Terence, > >>>> Im running it on a non-secure cluster. Im launching the job as my own > >> user > >>>> (say.. sam_william), I can see the job on the web UI, and it shows the > >>>> correct user id (sam_william). > >>>> That brings us to another point, that I forgot to add earlier. Im > >>>> actually trying to create a file on HDFS from inside a > TwillRunnable. I > >>>> thought I could use getContext() to get a Configuration() object, but > >> it > >>>> looks like the context does not contain a reference to Configuration. > I > >>>> just went ahead and did a Filesystem.get(new Configuration()) and it > >>>> worked. I was able create and write into the file. When I check the > >> file > >>>> after the job is completed, I see that the file is owned by user > >> ‘yarn’ . > >>>> > >>>> Sam > >>>> > >>>> > >>>>> On Feb 22, 2016, at 11:54 AM, Terence Yim <[email protected]> wrote: > >>>>> > >>>>> Hi Sam, > >>>>> > >>>>> Are you running a non-secure cluster or a Kerberos cluster? And what > >> user > >>>>> you used to launch the job on the client side? > >>>>> In non-secure cluster, the container user would be the same as the > >> login > >>>>> user you used to launch the job. For secure cluster, the current user > >> in > >>>>> UserGroupInformation will be used (usually the Kerberos authenticated > >>>> user > >>>>> you are using to launch the job). > >>>>> > >>>>> Terence > >>>>> > >>>>> On Mon, Feb 22, 2016 at 11:29 AM, Sam Prince Daniel William < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> Hi, > >>>>>> I couldn't find user mailing list, hence asking this question here. > >> I > >>>>>> have been playing around with twill for the last few days and > >> wondering > >>>> if > >>>>>> there is a way to prevent the container tasks running as the > superuser > >>>>>> (yarn in my case) ? I tried using the preparer.setUser() method, but > >>>> that > >>>>>> didn't have any effect. > >>>>>> > >>>>>> Sam > >>>>>> > >>>> > >>>> > >> > >> > >
