Re: Job Submission to Cross Cluster Fails

[Peter Cseh]

Hi,

Have you tried running dryruns to see the exact configurations the
workflows and the actions are getting from Hue?
Oozie is designed to work against multiple clusters, but I don't know if it
can be done via Hue.

gp

On Mon, Mar 6, 2017 at 8:49 PM, mdk-swandha 
wrote:

> Hi,
>
> I have the following configuration:
>
> Cluster 1:
> Job Submission through Hue (hue1) to Cluster 1 (NN1 and RM1) via Oozie
> works.
>
> Cluster 2:
> Job Submission through Hue (hue2) to Cluster 2 (NN2 and RM2) via Oozie
> works.
>
> As Oozie is cluster agnostic - now I have tweaked my Hue code to submit job
> to cross cluster i.e. from Hue1 to Cluster2 (NN2 and RM2). In the workspace
> folder I do see job.properties with modified NN and JT (RM)
>
> Job is going to RM2 but it is failing there with the following warnings and
> error
>
> Warning:
>
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> final parameter: fs.defaultFS;  Ignoring.
> org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
> final parameter: fs.defaultFS;  Ignoring.
>
>
> java.io.FileNotFoundException: File does not exist:
> hdfs://cluster1-nn1:8020/user/user1/.staging/job_1488573599716_0001/job.
> splitmetainfo
>
>
> org.apache.hadoop.security.token.SecretManager$InvalidToken):
> appattempt_1488573599716_0001_01 not found in
> AMRMTokenSecretManager.
>
>
> Also it is trying to connect to RM1 as it shows the following message in
> the log
>
>
> INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to
> ResourceManager at RM1/xx.xx.xx.xx:8030
>
>
> I have modified following in Oozie-site.xml
>
>
> oozie.service.ProxyUserService.proxyuser.hue.groups ==> *
>
> oozie.service.ProxyUserService.proxyuser.hue.hosts ==> *
>
>
> I would appreciate if someone can shed some light on how to make this work.
>
>
> Do I need to route job to cluster specific Oozie server?
>
>
> Why job.xml (sent to RM via Oozie) has Cluster-1's NN and RM
> information (if that is the case)?
>
>
> Do I require any other configuration setting to enable this cross
> cluster job submission work?
>
>
> Thanks.
>



-- 
Peter Cseh
Software Engineer

Job Submission to Cross Cluster Fails

[mdk-swandha]

Hi,

I have the following configuration:

Cluster 1:
Job Submission through Hue (hue1) to Cluster 1 (NN1 and RM1) via Oozie
works.

Cluster 2:
Job Submission through Hue (hue2) to Cluster 2 (NN2 and RM2) via Oozie
works.

As Oozie is cluster agnostic - now I have tweaked my Hue code to submit job
to cross cluster i.e. from Hue1 to Cluster2 (NN2 and RM2). In the workspace
folder I do see job.properties with modified NN and JT (RM)

Job is going to RM2 but it is failing there with the following warnings and
error

Warning:

org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
final parameter: fs.defaultFS;  Ignoring.
org.apache.hadoop.conf.Configuration: job.xml:an attempt to override
final parameter: fs.defaultFS;  Ignoring.


java.io.FileNotFoundException: File does not exist:
hdfs://cluster1-nn1:8020/user/user1/.staging/job_1488573599716_0001/job.splitmetainfo


org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1488573599716_0001_01 not found in
AMRMTokenSecretManager.


Also it is trying to connect to RM1 as it shows the following message in the log


INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to
ResourceManager at RM1/xx.xx.xx.xx:8030


I have modified following in Oozie-site.xml


oozie.service.ProxyUserService.proxyuser.hue.groups ==> *

oozie.service.ProxyUserService.proxyuser.hue.hosts ==> *


I would appreciate if someone can shed some light on how to make this work.


Do I need to route job to cluster specific Oozie server?


Why job.xml (sent to RM via Oozie) has Cluster-1's NN and RM
information (if that is the case)?


Do I require any other configuration setting to enable this cross
cluster job submission work?


Thanks.