[
https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170486#comment-13170486
]
Anupam Seth commented on MAPREDUCE-3251:
----------------------------------------
bq. Anupam, did you do a real cluster-test or an integration test?
@Vinod, yes I did. Here is console output upon disabling the ACL and running a
word count job. I think I see the intent of what you are saying, and it will
probably definitely be cleaner, but for some reason, it isn't as broken I think.
@Mahadev, I will upload a new patch with the suggestions you have outlined in
pursuance of the above comments.
11/12/15 20:41:32 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
11/12/15 20:41:32 WARN conf.Configuration: mapred.used.genericoptionsparser is
deprecated. Instead, use mapreduce.client.genericoptionsparser.used
11/12/15 20:41:32 INFO input.FileInputFormat: Total input paths to process : 1
11/12/15 20:41:32 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
11/12/15 20:41:32 WARN snappy.LoadSnappy: Snappy native library not loaded
11/12/15 20:41:32 INFO mapreduce.JobSubmitter: number of splits:1
11/12/15 20:41:33 INFO mapred.ResourceMgrDelegate: Submitted application
application_1323981651676_0001 to ResourceManager at
<hostname>/98.139.92.65:8040
11/12/15 20:41:33 INFO mapreduce.Job: Running job: job_1323981651676_0001
11/12/15 20:41:42 INFO mapred.ClientServiceDelegate: AppId:
application_1323981651676_0001 # reserved containers: 0 # used containers: 1
Needed resources (memory): 2048 Reserved resources (memory): 0 Used resources
(memory): 2048 Diagnostics: Start time: 1323981693246 Finish time: 0 Host:
<hostname> Name: word count Orig. tracking url: <hostname>:50256 Queue: default
RPC port: 55191 Tracking url:
<hostname>:8088/proxy/application_1323981651676_0001/ User: <user> Client
token: null Final appl. status: UNDEFINED Yarn appl. state: RUNNING
....
....
....
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: Network ACL closed to AM
for job job_1323981651676_0001. Redirecting to job history server.
11/12/15 20:41:56 WARN mapred.ClientServiceDelegate: Job History Server is not
configured or job information not yet available on History Server.
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: AppId:
application_1323981651676_0001 # reserved containers: 0 # used containers: 1
Needed resources (memory): 2048 Reserved resources (memory): 0 Used resources
(memory): 2048 Diagnostics: Start time: 1323981693246 Finish time: 0 Host:
<hostname> Name: word count Orig. tracking url: <hostname>:50256 Queue: default
RPC port: 55191 Tracking url:
<hostname>:8088/proxy/application_1323981651676_0001/ User: <user> Client
token: null Final appl. status: UNDEFINED Yarn appl. state: RUNNING
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: Network ACL closed to AM
for job job_1323981651676_0001. Redirecting to job history server.
11/12/15 20:41:56 WARN mapred.ClientServiceDelegate: Job History Server is not
configured or job information not yet available on History Server.
11/12/15 20:41:57 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
11/12/15 20:41:57 WARN mapred.ClientServiceDelegate: Job History Server is not
configured or job information not yet available on History Server.
11/12/15 20:41:57 INFO mapreduce.Job: Job job_1323981651676_0001 completed
successfully
11/12/15 20:41:57 INFO mapreduce.Job: Counters: 0
> Network ACLs can prevent some clients to talk to MR ApplicationMaster
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-3251
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3251-branch_0_23.patch,
> MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251-branch_0_23.patch,
> MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251_branch-0_23_preliminary.txt
>
>
> In 0.20.xxx, the JobClient while polling goes to JT to get the job status.
> With YARN, AM can be launched on any port and the client will have to have
> ACL open to that port to talk to AM and get the job status. When the client
> is within the same grid network access to AM is not a problem. But some
> applications may have one installation per set of clusters and may launch
> jobs even across such sets (on job trackers in another set of clusters). For
> that to work only the JT port needs to be open currently. In case of YARN,
> all ports will have to be opened up for things to work. That would be a
> security no-no.
> There are two possible solutions:
> 1) Make the job client only talk to RM (as an option) to get the job
> status.
> 2) Limit the range of ports AM can listen on.
> Option 2) may not be favorable as there is no direct OS API to find a free
> port.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
