The doAs method in UserGroupInformation is what you want when dealing with
multiple UGIs. It determines what UGI instance the code within the doAs scope
gets when that code tries to lookup the current user.
Each AM is designed to run in a separate JVM, so each has some main()-like
entry point that does everything to setup the AM. Theoretically all you need
to do is create two, separate UGIs then use each instance to perform a doAs
wrapping the invocation of the corresponding AM's entry point. After that,
everything that AM does will get the UGI of the doAs invocation as the current
user. Since the AMs are running in separate doAs instances they will get
separate UGIs for the current user and thus separate credentials.
Jason
On Thursday, March 16, 2017 4:03 PM, Sergiy Matusevych
<[email protected]> wrote:
Hi Jason,
Thanks a lot for your help again! Having two separate UserGroupInformation
instances is exactly what I had in mind. What I do not understand, though, is
how to make sure that our second call to .regsiterApplicationMaster() will pick
the right UserGroupInformation object. I would love to find a way that does not
involve any changes to the YARN client, but if we have to patch it, of course,
I agree that we need to have a generic yet minimally invasive solution.
Thank you!Sergiy.
On Thu, Mar 16, 2017 at 8:03 AM, Jason Lowe <[email protected]> wrote:
>
> I believe a cleaner way to solve this problem is to create two, _separate_
> UserGroupInformation objects and wrap each AM instances in a UGI doAs so they
> aren't trying to share the same credentials. This is one example of a token
> bleeding over and causing problems. I suspect trying to fix these one-by-one
> as they pop up is going to be frustrating compared to just ensuring the
> credentials remain separate as if they really were running in separate JVMs.
>
> Adding Daryn who knows a lot more about the UGI stuff so he can correct any
> misunderstandings on my part.
>
> Jason
>
>
> On Wednesday, March 15, 2017 1:11 AM, Sergiy Matusevych
> <[email protected]> wrote:
>
>
> Hi YARN developers,
>
> I have an interesting problem that I think is related to YARN Java client.
> I am trying to launch *two* application masters in one container. To be
> more specific, I am starting a Spark job on YARN, and launch an Apache REEF
> Unmanaged AM from the Spark Driver.
>
> Technically, YARN Resource Manager should not care which process each AM
> runs in. However, there is a problem with the YARN Java client
> implementation: there is a global UserGroupInformation object that holds
> the user credentials of the current RM session. This data structure is
> shared by all AMs, and when REEF application tries to register the second
> (unmanaged) AM, the client library presents to YARN RM all credentials,
> including the security token of the first (managed) AM. YARN rejects such
> registration request, throwing InvalidApplicationMasterRequestException
> "Application Master is already registered".
>
> I feel like this issue can be resolved by a relatively small update to the
> YARN Java client - e.g. by introducing a new variant of the
> AMRMClientAsync.registerApplicationMaster() that would take the required
> security token (instead of getting it implicitly from the
> UserGroupInformation.getCurrentUser().getCredentials() etc.), or having
> some sort of RM session class that would wrap all data that is currently
> global. I need to think about the elegant API for it.
>
> What do you guys think? I would love to work on this problem and send you a
> pull request for the upcoming 2.9 release.
>
> Cheers,
> Sergiy.
>
>
