[jira] [Commented] (MAPREDUCE-7131) Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate
[
https://issues.apache.org/jira/browse/MAPREDUCE-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594688#comment-16594688
]
Varun Saxena commented on MAPREDUCE-7131:
-
[~erwaman], added you to the list of contributors and assigned the JIRA to you
> Job History Server has race condition where it moves files from intermediate
> to finished but thinks file is in intermediate
> ---
>
> Key: MAPREDUCE-7131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
>Priority: Major
>
> This is the race condition that can occur:
> # during the first *scanIntermediateDirectory()*,
> *HistoryFileInfo.moveToDone()* is scheduled for job j1
> # during the second *scanIntermediateDirectory()*, j1 is found again and put
> in the *fileStatusList* to process
> # *HistoryFileInfo.moveToDone()* is processed in another thread and history
> files are moved to the finished directory
> # the *HistoryFileInfo* for j1 is removed from *jobListCache*
> # the j1 in *fileStatusList* is processed and a new *HistoryFileInfo* for j1
> is created (history, conf, and summary files will point to the intermediate
> user directory, and state will be IN_INTERMEDIATE)
> # *moveToDone()* is scheduled for this new j1
> # *moveToDone()* fails during *moveToDoneNow()* for the history file because
> the source path in the intermediate directory does not exist
> From this point on, while the new j1 *HistoryFileInfo* is in the
> *jobListCache*, the JobHistoryServer will think the history file is in the
> intermediate directory. If a user queries this job in the JobHistoryServer
> UI, they will get
> {code}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load
> history file
> ://:/mr-history/intermediate//job_1529348381246_27275711-1535123223269---1535127026668-1-0-SUCCEEDED--1535126980787.jhist
> {code}
> Noticed this issue while running 2.7.4, but the race condition seems to still
> exist in trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7131) Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate
[
https://issues.apache.org/jira/browse/MAPREDUCE-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena reassigned MAPREDUCE-7131:
---
Assignee: Anthony Hsu
> Job History Server has race condition where it moves files from intermediate
> to finished but thinks file is in intermediate
> ---
>
> Key: MAPREDUCE-7131
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7131
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.4
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
>Priority: Major
>
> This is the race condition that can occur:
> # during the first *scanIntermediateDirectory()*,
> *HistoryFileInfo.moveToDone()* is scheduled for job j1
> # during the second *scanIntermediateDirectory()*, j1 is found again and put
> in the *fileStatusList* to process
> # *HistoryFileInfo.moveToDone()* is processed in another thread and history
> files are moved to the finished directory
> # the *HistoryFileInfo* for j1 is removed from *jobListCache*
> # the j1 in *fileStatusList* is processed and a new *HistoryFileInfo* for j1
> is created (history, conf, and summary files will point to the intermediate
> user directory, and state will be IN_INTERMEDIATE)
> # *moveToDone()* is scheduled for this new j1
> # *moveToDone()* fails during *moveToDoneNow()* for the history file because
> the source path in the intermediate directory does not exist
> From this point on, while the new j1 *HistoryFileInfo* is in the
> *jobListCache*, the JobHistoryServer will think the history file is in the
> intermediate directory. If a user queries this job in the JobHistoryServer
> UI, they will get
> {code}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not load
> history file
> ://:/mr-history/intermediate//job_1529348381246_27275711-1535123223269---1535127026668-1-0-SUCCEEDED--1535126980787.jhist
> {code}
> Noticed this issue while running 2.7.4, but the race condition seems to still
> exist in trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466353#comment-16466353
]
Varun Saxena commented on MAPREDUCE-6826:
-
[~BilwaST], thanks for the patch.
Can you fix the errors due to tabs?
Also can you add a test case for COMMITTING-> JOB_TASK_COMPLETED transition as
well by using WaitingOutputCommitter.
Also name the test case method in a way that it reflects the transition being
tested. Something like TestJobTaskCompletedWhileCommitting, for instance.
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
>
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch, MAPREDUCE-6826-002.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0
[jira] [Comment Edited] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466353#comment-16466353
]
Varun Saxena edited comment on MAPREDUCE-6826 at 5/7/18 7:28 PM:
-
[~BilwaST], thanks for the patch.
Can you fix the errors due to tabs?
Also can you add a test case for COMMITTING-> JOB_TASK_COMPLETED transition as
well by using WaitingOutputCommitter.
Additionally, name the test case method in a way that it reflects the
transition being tested. Something like TestJobTaskCompletedWhileCommitting,
for instance.
was (Author: varun_saxena):
[~BilwaST], thanks for the patch.
Can you fix the errors due to tabs?
Also can you add a test case for COMMITTING-> JOB_TASK_COMPLETED transition as
well by using WaitingOutputCommitter.
Also name the test case method in a way that it reflects the transition being
tested. Something like TestJobTaskCompletedWhileCommitting, for instance.
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
>
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch, MAPREDUCE-6826-002.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
>
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6826:
Summary: Job fails with InvalidStateTransitonException: Invalid event:
JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING (was: Job fails with
InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED)
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
>
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch, MAPREDUCE-6826-002.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436812#comment-16436812
]
Varun Saxena commented on MAPREDUCE-6826:
-
The build is not invoking Mapreduce tests. Although change is trivial, can you
add a small test in the patch to verify the fix? That may invoke the build.
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
>
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435173#comment-16435173
]
Varun Saxena commented on MAPREDUCE-6826:
-
Running build again to invoke MR unit tests
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433746#comment-16433746
]
Varun Saxena commented on MAPREDUCE-6826:
-
Submitting patch to invoke QA
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23 09:10:38,993 INFO
[jira] [Commented] (MAPREDUCE-6332) Provide facility to users for writting custom MergeManager implementation when custom shuffleconsumerPluggin is used
[ https://issues.apache.org/jira/browse/MAPREDUCE-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400092#comment-16400092 ] Varun Saxena commented on MAPREDUCE-6332: - [~rohithsharma] , are we planning to get this JIRA in? > Provide facility to users for writting custom MergeManager implementation > when custom shuffleconsumerPluggin is used > > > Key: MAPREDUCE-6332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6332 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: 0001-MAPREDUCE-6332.patch, 0002-MAPREDUCE-6332.patch, > 0003-MAPREDUCE-6332.patch, 0004-MAPREDUCE-6332.patch > > > MR provides ability to the user for plugin custom ShuffleConsumerPlugin using > *mapreduce.job.reduce.shuffle.consumer.plugin.class*. When the user is > allowed to use this configuration as plugin, user also interest in > implementing his own MergeManagerImpl. > But now , user is forced to use MR provided MergeManagerImpl instead of > custom MergeManagerImpl when user is using shuffle.consumer.plugin class. > There should be well defined API's in MergeManager that can be used for any > implementation without much effort to user for custom implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena resolved MAPREDUCE-6732.
-
Resolution: Fixed
Hadoop Flags: Reviewed
> mapreduce tasks for YARN Timeline Service v.2: alpha 2
> --
>
> Key: MAPREDUCE-6732
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Sangjin Lee
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-beta1
>
>
> This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline
> Service v.2 alpha 2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6331) [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928)
[ https://issues.apache.org/jira/browse/MAPREDUCE-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6331: Fix Version/s: 2.9.0 > [Umbrella] Make MapReduce work with Timeline Service Nextgen (YARN-2928) > > > Key: MAPREDUCE-6331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6331 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > > Tracking umbrella for all MR changes to make it work with Timeline Service > Nextgen - YARN-2928. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6732) mapreduce tasks for YARN Timeline Service v.2: alpha 2
[
https://issues.apache.org/jira/browse/MAPREDUCE-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6732:
Fix Version/s: 3.0.0-beta1
2.9.0
> mapreduce tasks for YARN Timeline Service v.2: alpha 2
> --
>
> Key: MAPREDUCE-6732
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6732
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Reporter: Sangjin Lee
>Assignee: Vrushali C
> Fix For: 2.9.0, 3.0.0-beta1
>
>
> This s an umbrella JIRA to capture all mapreduce tasks for YARN Timeline
> Service v.2 alpha 2.
> This is developed on feature branches: {{YARN-5355}} for the trunk-based
> development and {{YARN-5355-branch-2}} to maintain backports to branch-2. Any
> subtask work on this JIRA will be committed to those 2 branches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Fix Version/s: 2.9.0 > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, 3.0.0-beta1, YARN-5355, YARN-5355-branch-2 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6688) Store job configurations in Timeline Service v2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6688: Fix Version/s: 2.9.0 > Store job configurations in Timeline Service v2 > --- > > Key: MAPREDUCE-6688 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6688 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6688-YARN-2928.01.patch, > MAPREDUCE-6688-YARN-2928.02.patch, MAPREDUCE-6688-YARN-2928.03.patch, > MAPREDUCE-6688-YARN-2928.04.patch, MAPREDUCE-6688-YARN-2928.v2.01.patch, > MAPREDUCE-6688-YARN-2928.v2.02.patch, YARN-3959-YARN-2928.01.patch > > > We already have configuration field in HBase schema for application entity. > We need to make sure AM write it out when it get launched. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6546) reconcile the two versions of the timeline service performance tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6546: Fix Version/s: 2.9.0 > reconcile the two versions of the timeline service performance tests > > > Key: MAPREDUCE-6546 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6546 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6546-YARN-2928.01.patch, > MAPREDUCE-6546-YARN-2928.02.patch, MAPREDUCE-6546-YARN-2928.03.patch > > > The trunk now has a version of the timeline service performance test > (YARN-2556). The timeline service v.2 (YARN-2928) also has a performance > test, and these two versions are quite similar (by design). > We need to reconcile the two. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6720) Inconsistent values of counters across tasks and job reported to timeline service.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6720: Fix Version/s: 2.9.0 > Inconsistent values of counters across tasks and job reported to timeline > service. > -- > > Key: MAPREDUCE-6720 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6720 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Counters For Job.png, Counters for Tasks.png, > MAPREDUCE-6720-YARN-2928.01.patch, MAPREDUCE-6720-YARN-2928.02.patch, > MAPREDUCE-6720-YARN-2928.03.patch, MAPREDUCE-6720-YARN-2928.04.patch > > > While testing found below issue. For some of the task counters, we do not > have consistent values. This is not the case with every counter though. > Consider the case of counter > "org.apache.hadoop.mapreduce.FileSystemCounter:FILE_BYTES_WRITTEN". > I found that its value for a flow I ran, was 936018 bytes. For the 3 apps > associated with this flow run, the values were 312006 bytes each (which > equals to value for a flow run i.e. 3 * 312006 = 936018). Drilling further > down I found though that for one of the apps, the 4 tasks(2 mappers and 2 > reducers) had values as 155918 bytes each for the 2 reducers and 156003 bytes > each for the 2 mappers. > This means the value reported for the app should be (2 * 156003 + 2* 155918) > or 623842 bytes but it is only 312006 bytes which indicates that only counter > value of mappers is being picked up. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6370) Timeline service v2 load generator needs to write event id
[ https://issues.apache.org/jira/browse/MAPREDUCE-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6370: Fix Version/s: 2.9.0 > Timeline service v2 load generator needs to write event id > -- > > Key: MAPREDUCE-6370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3702-YARN-2928.001.patch > > > We need to write a sample event id in SimpleEntityWriter so that both HBase > and Phoenix writers can actually write the timeline event. For now the > Phoenix implementation will throw exceptions and the HBase will skip storing > the timeline event. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6424: Fix Version/s: 2.9.0 > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch, > MAPREDUCE-6424-YARN-2928.v1.002.patch, MAPREDUCE-6424-YARN-2928.v1.003.patch, > MAPREDUCE-6424-YARN-2928.v1.004.patch, MAPREDUCE-6424-YARN-2928.v1.005.patch > > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6337) add a mode to replay MR job history files to the timeline service
[ https://issues.apache.org/jira/browse/MAPREDUCE-6337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6337: Fix Version/s: 2.9.0 > add a mode to replay MR job history files to the timeline service > - > > Key: MAPREDUCE-6337 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6337 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6337-YARN-2928.001.patch, > MAPREDUCE-6337-YARN-2928.002.patch, MAPREDUCE-6337-YARN-2928.003.patch, > MAPREDUCE-6337-YARN-2928.004.patch > > > The subtask covers the work on top of YARN-3437 to add a mode to replay MR > job history files to the timeline service storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance
[ https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6372: Fix Version/s: 2.9.0 > clean up several issues with TimelineServicePerformance > --- > > Key: MAPREDUCE-6372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > > We found a few issues with the TimelineServicePerformanceV2 test driver while > running it for the performance tests. Filing this JIRA to fix those issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6335) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6335: Fix Version/s: 2.9.0 > convert load test driver to timeline service v.2 > > > Key: MAPREDUCE-6335 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6335 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6335.005.patch, MAPREDUCE-6335.006.patch, > YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, > YARN-3437.004.patch > > > This subtask covers the work for converting the proposed patch for the load > test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6327) [Event producers] Implement MapReduce AM writing MR events/counters to v2 ATS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6327: Fix Version/s: 2.9.0 > [Event producers] Implement MapReduce AM writing MR events/counters to v2 ATS > - > > Key: MAPREDUCE-6327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6327 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Sangjin Lee >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, > YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, > YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch, YARN-3046-v6.patch > > > Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes > written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6818: Fix Version/s: 2.9.0 > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Fix For: 2.9.0, 3.0.0-beta1, YARN-5355 > > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Fix Version/s: 3.0.0-beta1 > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 3.0.0-beta1, YARN-5355, YARN-5355-branch-2 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6818) Remove direct reference to TimelineClientImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6818: Fix Version/s: 3.0.0-beta1 > Remove direct reference to TimelineClientImpl > - > > Key: MAPREDUCE-6818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6818 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Li Lu >Assignee: Li Lu > Labels: newbie++ > Fix For: 3.0.0-beta1, YARN-5355 > > Attachments: YARN-5974-YARN-5355.001.patch > > > [~sjlee0]'s quick audit shows that things that are referencing > TimelineClientImpl directly today: > JobHistoryFileReplayMapperV1 (MR) > SimpleEntityWriterV1 (MR) > TestDistributedShell (DS) > TestDSAppMaster (DS) > TestNMTimelinePublisher (node manager) > TestTimelineWebServicesWithSSL (AHS) > This is not the right way to use TimelineClient and we should avoid direct > reference to TimelineClientImpl as much as possible. > Any newcomers to the community are more than welcome to take this. If this > remains unassigned for ~24hrs I'll jump in and do a quick fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Fix Version/s: YARN-5355-branch-2 > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355, YARN-5355-branch-2 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136379#comment-16136379 ] Varun Saxena commented on MAPREDUCE-6838: - Cherry picked MAPREDUCE-6838 to YARN-5355_branch2. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136345#comment-16136345 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~jianhe] for the review and commit. Thanks [~rohithsharma] for reviews. I will resolve branch2 compilation issue and then cherry-pick this as well. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136206#comment-16136206 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/22/17 3:41 AM: -- Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish existing entities sitting in queue. As there can be a potential race, so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. Not sure why. Should we add one? was (Author: varun_saxena): Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish. So there can be a potential race so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. Not sure why. Should we add one? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136206#comment-16136206 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/22/17 3:37 AM: -- Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish. So there can be a potential race so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. Not sure why. Should we add one? was (Author: varun_saxena): Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish. So there can be a potential race so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136206#comment-16136206 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~jianhe] for the comments. bq. The comment says is OR condition where as the code is AND, which one is true? The code condition is correct. Will change the comment. bq. Also, when will the "delegationToken.getService()" be empty ? These are just checks for sanity. As NodeTimelineCollectorManager belongs to timelineservice module and this to yarn-common. So added these checks because change elsewhere should not break code here. bq. it uses "SecurityUtil.getTokenServiceAddr(timelineToken)" to set the token service. Then next time collectorAddr is not null because timelineServiceAddress is not null, it always call "NetUtils.createSocketAddr(collectorAddr) " to set the token service. Is my understanding correct? why not just consistently use one of them to make it look simpler? So this is because we are polling on timelineservice address in another thread(entity dispatcher) and as soon as it is found, we go on to publish. So there can be a potential race so I first update the token and then the timeline address. I can write a comment in code to make this clear. bq. Does the collector address change if NM restarts? If so, we may have two keys(different address) for two tokens in the UGI. Yes, that's true but the token will be picked up by DelegationTokenAuthenticatedURL based on current collector address. Could not find any API to remove the token from UGI. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135464#comment-16135464 ] Varun Saxena commented on MAPREDUCE-6838: - Test failures are unrelated. They are outstanding issues on trunk > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134420#comment-16134420 ] Varun Saxena commented on MAPREDUCE-6838: - Attaching a patch which makes token variable volatile, adds a LOG when collectorinfo is null and does small refactoring. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Open (was: Patch Available) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: Open) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.06.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch, > MAPREDUCE-6838-YARN-5355.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134241#comment-16134241 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/19/17 9:23 PM: -- Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token, it is picked from UGI. Will it be likely that token will be updated from 2 separate threads? Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. Infact it helps us handle cases when token is updated from 2 threads(if any such situation arises). We can make it volatile I guess. bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not required since equals does this comparrission too. Didn't quite get you. This is to avoid updating token for another kind. This is to avoid updating a token altogether i.e. even if we do not have a previous token. The equals check is for not updating the token if it is equal to cached token. If I remove this check, a token of another kind will be added in UGI. bq. In CollectorInfo object, If collector address is null and Token is non-null. Do not add that token into ugi. Check like this required? If token service exists and timeline service address is already updated, should we not update the token, if we look at this piece of code independently. Currently we send both together but the protocol doesn't enforce it. The proto definition of CollectorInfo marks collector address field as optional. These checks are primarily for robustness if we consider the TimelineV2Client code in isolation and not merely go by what we currently know RM sends. If we make assumptions based on current implementation, we are tightly coupling the RM/NM logic with logic here and it is not enforced by protocol either. It is likely to work just fine as developers would take care but I would suggest that ideally if we assume that collector address is carried always, we enforce it in proto definition of CollectorInfo i.e. make collector address as "required" instead of "optional" in it. And we will have to see if address should be "required" in AppCollectorData too. Please note that RM may not have access to collector address initially when AM container is launched. So this change would also mean change in RM to not send collector info at all if address is null. The last comment i.e. suggested refactoring depends on comments above i.e. whether to update the token or not if address is not carried in collector info. Thoughts? was (Author: varun_saxena): Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token, it is picked from UGI. Will it be likely that token will be updated from 2 separate threads? We anyways do not claim any thread safety for timeline client. Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. As you say. I do not have a strong opinion on this. Thoughts? bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134241#comment-16134241 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/19/17 8:36 PM: -- Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token, it is picked from UGI. Will it be likely that token will be updated from 2 separate threads? We anyways do not claim any thread safety for timeline client. Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. As you say. I do not have a strong opinion on this. Thoughts? bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not required since equals does this comparrission too. Didn't quite get you. This is to avoid updating token for another kind. This is to avoid updating a token altogether i.e. even if we do not have a previous token. The equals check is for not updating the token if it is equal to cached token. If I remove this check, a token of another kind will be added in UGI. bq. In CollectorInfo object, If collector address is null and Token is non-null. Do not add that token into ugi. Check like this required? If token service exists and timeline service address is already updated, should we not update the token, if we look at this piece of code independently. Currently we send both together but the protocol doesn't enforce it. The proto definition of CollectorInfo marks collector address field as optional. These checks are primarily for robustness if we consider the TimelineV2Client code in isolation and not merely go by what we currently know RM sends. If we make assumptions based on current implementation, we are tightly coupling the RM/NM logic with logic here and it is not enforced by protocol either. It is likely to work just fine as developers would take care but I would suggest that ideally if we assume that collector address is carried always, we enforce it in proto definition of CollectorInfo i.e. make collector address as "required" instead of "optional" in it. And we will have to see if address should be "required" in AppCollectorData too. Please note that RM may not have access to collector address initially when AM container is launched. So this change would also mean change in RM to not send collector info at all if address is null. The last comment i.e. suggested refactoring depends on comments above i.e. whether to update the token or not if address is not carried in collector info. Thoughts? was (Author: varun_saxena): Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token is picked from UGI. Will it be likely that token will be updated from 2 separate threads? We anyways do not claim any thread safety for timeline client. Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. As you say. I do not have a strong opinion on this. Thoughts? bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not required since
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134241#comment-16134241 ] Varun Saxena commented on MAPREDUCE-6838: - Thanks [~rohithsharma] for the review. bq. Need to log a WARN message if collector info is null. Ok. bq. currentTimelineToken should be volatile Need not be. Atleast in MR AM. This is used only while AM is updating the token and that happens only from RMContainer Allocator thread so only one thread sees and updates it. While using token is picked from UGI. Will it be likely that token will be updated from 2 separate threads? We anyways do not claim any thread safety for timeline client. Address is volatile and its different because the thread publishing the entity and using the address would be different from the allocator thread which would communicate with RM and update the address. However, making it volatile doesn't cost us anything. As you say. I do not have a strong opinion on this. Thoughts? bq. Creating Token does not required to check service==null. Internally constructor does. And we can ignore token service passed by delegationToken always and set it up collector address. You mean the constructor inside setTimelineDelegationToken method i.e. at L203? Actually the constructor takes service as Text and not String. The check I am making is for service as String. If I do not make the check and call new Text(service), a null service would throw NPE. bq. !delegationToken.getKind().equals(TimelineDelegationTokenIdentifier.KIND_NAME.toString()) check is not required since equals does this comparrission too. Didn't quite get you. This is to avoid updating token for another kind. This is to avoid updating a token altogether i.e. even if we do not have a previous token. The equals check is for not updating the token if it is equal to cached token. If I remove this check, a token of another kind will be added in UGI. bq. In CollectorInfo object, If collector address is null and Token is non-null. Do not add that token into ugi. Check like this required? If token service exists and timeline service address is already updated, should we not update the token, if we look at this piece of code independently. Currently we send both together but the protocol doesn't enforce it. The proto definition of CollectorInfo marks collector address field as optional. These checks are primarily for robustness if we consider the TimelineV2Client code in isolation and not merely go by what we currently know RM sends. If we make assumptions based on current implementation, we are tightly coupling the RM/NM logic with logic here and it is not enforced by protocol either. It is likely to work just fine as developers would take care but I would suggest that ideally if we assume that collector address is carried always, we enforce it in proto definition of CollectorInfo i.e. make collector address as "required" instead of "optional" in it. And we will have to see if address should be "required" in AppCollectorData too. Please note that RM may not have access to collector address initially when AM container is launched. So this change would also mean change in RM to not send collector info at all if address is null. The last comment i.e. suggested refactoring depends on comments above i.e. whether to update the token or not if address is not carried in collector info. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6935) Allow multiple active timeline clients
[ https://issues.apache.org/jira/browse/MAPREDUCE-6935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6935: Parent Issue: MAPREDUCE-6943 (was: MAPREDUCE-6732) > Allow multiple active timeline clients > --- > > Key: MAPREDUCE-6935 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6935 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Aaron Gresch > > In order to migrate smoothly from timeline service v1 to v2, it would be > useful to be able to run both services at the same time for a period of time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6318) Refactor JobHistoryEventHandler for handling events in ATS v1
[ https://issues.apache.org/jira/browse/MAPREDUCE-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6318: Parent Issue: MAPREDUCE-6943 (was: MAPREDUCE-6732) > Refactor JobHistoryEventHandler for handling events in ATS v1 > - > > Key: MAPREDUCE-6318 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6318 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Reporter: Junping Du >Assignee: Junping Du > > Per discussion in YARN-3046, we need to refactor the process flow for > timeline events (ATS v1) in JobHistoryEventHandler. The refactor work should > get happen after YARN-2928 merged into trunk and branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6318) Refactor JobHistoryEventHandler for handling events in ATS v1
[ https://issues.apache.org/jira/browse/MAPREDUCE-6318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6318: Summary: Refactor JobHistoryEventHandler for handling events in ATS v1 (was: Refactor JobHistoryEventHandler for handling events in ATS v1 after YARN-2928 merge back to trunk and branch-2 ) > Refactor JobHistoryEventHandler for handling events in ATS v1 > - > > Key: MAPREDUCE-6318 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6318 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Reporter: Junping Du >Assignee: Junping Du > > Per discussion in YARN-3046, we need to refactor the process flow for > timeline events (ATS v1) in JobHistoryEventHandler. The refactor work should > get happen after YARN-2928 merged into trunk and branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6943) Mapreduce tasks for YARN Timeline Service v.2: beta 1
Varun Saxena created MAPREDUCE-6943: --- Summary: Mapreduce tasks for YARN Timeline Service v.2: beta 1 Key: MAPREDUCE-6943 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6943 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Varun Saxena Assignee: Varun Saxena -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: Open) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.05.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Open (was: Patch Available) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch, MAPREDUCE-6838-YARN-5355.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: Open) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.04.patch Updated the patch. Need to move out YARN changes to a separate YARN JIRA? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Open (was: Patch Available) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134021#comment-16134021 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/19/17 10:20 AM: --- Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token service. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? was (Author: varun_saxena): Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134029#comment-16134029 ] Varun Saxena commented on MAPREDUCE-6838: - bq. The client can construct the tokenService based on the collector address info ? I guess you were referring to timeline client too. I will update a patch by adding a new API in TimelineV2Client. We will use the token service if it comes in the token, otherwise use the address. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134021#comment-16134021 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/19/17 8:34 AM: -- Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? was (Author: varun_saxena): Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134021#comment-16134021 ] Varun Saxena commented on MAPREDUCE-6838: - Maybe what we can do is that provide another API in TimelineV2Client, say, setTimelineToken and that does the job of sanitizing the service and setting the token. That is, move the code from RMContainerAllocator to TimelineV2ClientImpl and set the token in UGI there. Thoughts? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134019#comment-16134019 ] Varun Saxena commented on MAPREDUCE-6838: - bq. today, for other delegation tokens RMDelegationToken, the old ATSv1 DelegationToken, the token service is not set at server side, it is set at client side - the client call the SecurityUtils#buildTokenService and then set the token service. I thought about this option too. But the issue I see here is that DelegationTokenAuthenticatedURL(used by TimelineV2ClientImpl) uses use_ip config and then each AM(if it does not use kerberos), will have to then write the code to sanitize the service coming in the token or fill the service based on collector address when token comes. This would not be done transparently. Currently we do this transparently for ATSv1 in YarnClientImpl > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: (was: MAPREDUCE-6838-YARN-5355.03.patch) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132092#comment-16132092 ] Varun Saxena commented on MAPREDUCE-6838: - bq. Why are changing hadoop-common code to ignore ignoreUseIpConfig? Is this because to add test case? Does test is failing? As discussed offline, this is to ensure that token service resolution is in sync between NM and AM. Also, with default configurations, E2E flow won't work. bq. I think it is better to update token on each heart beat rather then comparing each time. It avoids proto compare. This would involve iterating over all the tokens on each HB when we add token to UGI. Infact depending on how many tokens a AM has in UGI, this may be even more expensive. bq. This appears mixing collectorManager i.e NMCollectorService authentication and MR side. The latest patch file contains code from YARN-7006. I used a dummy patch just to ensure QA runs. Will delete that patch as YARN-7006 has gone in. bq. One thing I observe is TimelineClient is published using login user which is doAS. I think we should publish is as owner of application otherwise we end up in writing data into sub app application table as well. Fair point. We can raise a separate JIRA for this. And create a proxy user based on app user to ensure timeline client publishes with that. bq. I just noticed that TimelineDelegationTokenIdentifier#Renewer has methods renew/cancel. These creates V1 client. Does it going to be a problem? This is not used by us for V2 anywhere and the Renewer class is annotated as Private. I guess no client would explicitly call it and even if they do, it should fail. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128398#comment-16128398 ] Varun Saxena edited comment on MAPREDUCE-6838 at 8/16/17 4:07 PM: -- TestSecureMRTimelineEventHandling failure is strange. It passes for me everytime. Here, AM container is exiting with exit code 1. Ideallly LCE should be used but will be difficult to ensure a E2E test with it I guess. [~rohithsharma], can you reproduce the failure? was (Author: varun_saxena): TestSecureMRTimelineEventHandling failure is strange. It passes for me everytime. Here, AM container is exiting with exit code 1. Ideallly LCE should be used but will be difficult to ensure a E2E test with it I guess. [~rohithsharma], can you simulate the failure? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128398#comment-16128398 ] Varun Saxena commented on MAPREDUCE-6838: - TestSecureMRTimelineEventHandling failure is strange. It passes for me everytime. Here, AM container is exiting with exit code 1. Ideallly LCE should be used but will be difficult to ensure a E2E test with it I guess. [~rohithsharma], can you simulate the failure? > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch Thought that I will attach a patch which contains the previous patch and YARN-7006 patch as well so that QA can run. I will delete the patch once YARN-7006 is committed. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: In Progress) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch, > MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6838 started by Varun Saxena. --- > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127518#comment-16127518 ] Varun Saxena commented on MAPREDUCE-6838: - I have added a E2E test case with security enabled, in the patch. This is to primarily check if token goes all the way to AM and used by AM to publish entities. Have used a single kerberos principal for all the components, based on the current user running the test. Moreover, in AM, we use the job config which may not have same values of hadoop.security.token.service.use_ip config as NM which generates the token. In our deployments, we will keep this config same across both client and NM end but not sure if we can make an assumption. So, for ATSv2, in DelegationTokenAuthenticatedURL I have passed a flag to indicate if we can ignore this config while looking for token in UGI. At the NM end, we would not use the config to generate token service as well. Thoughts? The patch depends on YARN-7006 so not submitting it. > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: (was: MAPREDUCE-6838-YARN-5355.03.patch) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Open (was: Patch Available) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.03.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch, MAPREDUCE-6838-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.02.patch > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Status: Patch Available (was: In Progress) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch, > MAPREDUCE-6838-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [ATSv2 Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Summary: [ATSv2 Security] Add timeline delegation token received in allocate response to UGI (was: [Security] Add timeline delegation token received in allocate response to UGI) > [ATSv2 Security] Add timeline delegation token received in allocate response > to UGI > --- > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6838) [Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Labels: yarn-5355-merge-blocker (was: ) > [Security] Add timeline delegation token received in allocate response to UGI > - > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: MAPREDUCE-6838-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6836) exception thrown when accessing the job configuration web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948082#comment-15948082
]
Varun Saxena commented on MAPREDUCE-6836:
-
Thanks Haibo for the patch.
Javadoc warning is bogus.
Changes LGTM.
> exception thrown when accessing the job configuration web UI
>
>
> Key: MAPREDUCE-6836
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6836
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: webapps
>Affects Versions: 3.0.0-alpha2
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
>Priority: Minor
> Attachments: MAPREDUCE-6836.00.patch
>
>
> When I navigate the MR job web UI and click the configuration link, the AM
> shows an exception:
> {noformat}
> 2017-01-25 11:40:55,521 ERROR [qtp2126664214-26]
> org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /mapreduc
> e/conf/job_1485372765455_0002
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at
> com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1458)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
[jira] [Updated] (MAPREDUCE-6838) [Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6838: Attachment: MAPREDUCE-6838-YARN-5355.01.patch This patch is on top of YARN-5647, YARN-5648 and YARN-6130. > [Security] Add timeline delegation token received in allocate response to UGI > - > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: MAPREDUCE-6838-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-6838) [Security] Add timeline delegation token received in allocate response to UGI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6838 started by Varun Saxena. --- > [Security] Add timeline delegation token received in allocate response to UGI > - > > Key: MAPREDUCE-6838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Varun Saxena > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6404) Allow AM to specify a port range for starting its webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855672#comment-15855672 ] Varun Saxena commented on MAPREDUCE-6404: - Thanks [~djp] for the review and commit. > Allow AM to specify a port range for starting its webapp > > > Key: MAPREDUCE-6404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6404 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: MAPREDUCE-6404.01.patch, MAPREDUCE-6404.02.patch > > > Allow AM to specify a port range for starting its webapp -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6843) NM node count reported to AM in Allocate Response should consider node label partitions.
Varun Saxena created MAPREDUCE-6843: --- Summary: NM node count reported to AM in Allocate Response should consider node label partitions. Key: MAPREDUCE-6843 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6843 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6843) NM node count reported to AM in Allocate Response should consider requested node label partitions.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6843: Summary: NM node count reported to AM in Allocate Response should consider requested node label partitions. (was: NM node count reported to AM in Allocate Response should consider node label partitions.) > NM node count reported to AM in Allocate Response should consider requested > node label partitions. > -- > > Key: MAPREDUCE-6843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6843 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Varun Saxena >Assignee: Varun Saxena > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6404) Allow AM to specify a port range for starting its webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6404: Status: Patch Available (was: Open) > Allow AM to specify a port range for starting its webapp > > > Key: MAPREDUCE-6404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6404 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: MAPREDUCE-6404.01.patch, MAPREDUCE-6404.02.patch > > > Allow AM to specify a port range for starting its webapp -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6404) Allow AM to specify a port range for starting its webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6404: Attachment: MAPREDUCE-6404.02.patch > Allow AM to specify a port range for starting its webapp > > > Key: MAPREDUCE-6404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6404 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: MAPREDUCE-6404.01.patch, MAPREDUCE-6404.02.patch > > > Allow AM to specify a port range for starting its webapp -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6404) Allow AM to specify a port range for starting its webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6404: Status: Open (was: Patch Available) > Allow AM to specify a port range for starting its webapp > > > Key: MAPREDUCE-6404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6404 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: MAPREDUCE-6404.01.patch, MAPREDUCE-6404.02.patch > > > Allow AM to specify a port range for starting its webapp -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6404) Allow AM to specify a port range for starting its webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853677#comment-15853677 ] Varun Saxena commented on MAPREDUCE-6404: - Sure. Will fix and update the patch shortly. > Allow AM to specify a port range for starting its webapp > > > Key: MAPREDUCE-6404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6404 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: MAPREDUCE-6404.01.patch > > > Allow AM to specify a port range for starting its webapp -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6826:
Description:
This happens if a container is preempted by scheduler after job starts
committing.
And this exception in turn leads to application being marked as FAILED in YARN.
I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state is
COMMITTING or SUCCEEDED as job is in the process of finishing.
Also is there any point in attempting to scheduler another task attempt if job
is already in COMMITTING or SUCCEEDED state.
{noformat}
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from RUNNING to COMMITTING
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_REMOTE_CLEANUP for container
container_e55_1482404625971_23910_01_10 taskAttempt
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : linux-19:26009
2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the
event EventType: JOB_COMMIT
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
jvm_1482404625971_23910_m_60473139527690 asked for a task
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
JobFinishedEvent
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from COMMITTING to SUCCEEDED
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording
last MRAppMaster retry
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
isAMLastRetry: true
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
that shouldUnregistered is: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
services
2016-12-23 09:10:38,800 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
JobHistoryEventHandler. Size of the outstanding queue size is 1
2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed
container container_e55_1482404625971_23910_01_10
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1482404625971_23910_m_04_1: Container preempted by scheduler
2016-12-23 09:10:38,994 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_04_1 TaskAttempt Transitioned from SUCCEEDED
to KILLED
2016-12-23 09:10:38,995 INFO [AsyncDispatcher event handler]
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6826:
Description:
This happens if a container is preempted by scheduler after job starts
committing.
And this exception in turn leads to application being marked as FAILED in YARN.
I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state is
COMMITTING or SUCCEEDED as job is in the process of finishing.
{noformat}
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from RUNNING to COMMITTING
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_REMOTE_CLEANUP for container
container_e55_1482404625971_23910_01_10 taskAttempt
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : linux-19:26009
2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the
event EventType: JOB_COMMIT
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
jvm_1482404625971_23910_m_60473139527690 asked for a task
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
JobFinishedEvent
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from COMMITTING to SUCCEEDED
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording
last MRAppMaster retry
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
isAMLastRetry: true
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
that shouldUnregistered is: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
services
2016-12-23 09:10:38,800 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
JobHistoryEventHandler. Size of the outstanding queue size is 1
2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed
container container_e55_1482404625971_23910_01_10
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1482404625971_23910_m_04_1: Container preempted by scheduler
2016-12-23 09:10:38,994 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_04_1 TaskAttempt Transitioned from SUCCEEDED
to KILLED
2016-12-23 09:10:38,995 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_04 Task Transitioned from SUCCEEDED to
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6826:
Affects Version/s: 2.7.2
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
> CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772164#comment-15772164
]
Varun Saxena commented on MAPREDUCE-6826:
-
Yeah just updated the description.
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED
> -
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e55_1482404625971_23910_01_10
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
> CompletedMaps:5 CompletedReds:0 ContAlloc:8
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6826:
Description:
This happens if a container is preempted by scheduler after job starts
committing.
And this exception in turn leads to application being marked as FAILED in YARN.
I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state is
COMMITTING or SUCCEEDED as job is in the process of finishing.
{noformat}
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from RUNNING to COMMITTING
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_REMOTE_CLEANUP for container
container_e55_1482404625971_23910_01_10 taskAttempt
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
attempt_1482404625971_23910_m_04_1
2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : linux-19:26009
2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the
event EventType: JOB_COMMIT
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
jvm_1482404625971_23910_m_60473139527690 asked for a task
2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
JobFinishedEvent
2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1482404625971_23910Job
Transitioned from COMMITTING to SUCCEEDED
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording
last MRAppMaster retry
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
isAMLastRetry: true
2016-12-23 09:10:38,798 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
that shouldUnregistered is: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
2016-12-23 09:10:38,799 INFO [Thread-93]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
services
2016-12-23 09:10:38,800 INFO [Thread-93]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
JobHistoryEventHandler. Size of the outstanding queue size is 1
2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed
container container_e55_1482404625971_23910_01_10
2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
2016-12-23 09:10:38,993 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1482404625971_23910_m_04_1: Container preempted by scheduler
2016-12-23 09:10:38,994 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_04_1 TaskAttempt Transitioned from SUCCEEDED
to KILLED
2016-12-23 09:10:38,995 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_04 Task Transitioned from SUCCEEDED to
[jira] [Created] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
Varun Saxena created MAPREDUCE-6826: --- Summary: Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED Key: MAPREDUCE-6826 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Varun Saxena Assignee: Varun Saxena -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6801) Fix flaky TestKill.testKillJob()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677622#comment-15677622
]
Varun Saxena commented on MAPREDUCE-6801:
-
Committed to trunk, branch-2.
Thanks [~haibochen] for your contribution.
> Fix flaky TestKill.testKillJob()
>
>
> Key: MAPREDUCE-6801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: mapreduce6801.001.patch, mapreduce6801.002.patch
>
>
> TestKill.testKillJob often fails for the same reason with the following error
> message:
> {code}
> 1 tests failed.
> FAILED: org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob
> Error Message:
> Task state not correct expected: but was:
> Stack Trace:
> java.lang.AssertionError: Task state not correct expected: but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:84)
> {code}
> The root cause is that when the job is in KILLED state from an external view,
> TaskKillEvents and TaskAttemptKillEvents placed on the event loop queue may
> not have been processed by the dispatcher thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6801) Fix flaky TestKill.testKillJob()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6801:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
> Fix flaky TestKill.testKillJob()
>
>
> Key: MAPREDUCE-6801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: mapreduce6801.001.patch, mapreduce6801.002.patch
>
>
> TestKill.testKillJob often fails for the same reason with the following error
> message:
> {code}
> 1 tests failed.
> FAILED: org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob
> Error Message:
> Task state not correct expected: but was:
> Stack Trace:
> java.lang.AssertionError: Task state not correct expected: but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:84)
> {code}
> The root cause is that when the job is in KILLED state from an external view,
> TaskKillEvents and TaskAttemptKillEvents placed on the event loop queue may
> not have been processed by the dispatcher thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6801) Fix flaky TestKill.testKillJob()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676157#comment-15676157
]
Varun Saxena commented on MAPREDUCE-6801:
-
+1
Will commit it later today.
> Fix flaky TestKill.testKillJob()
>
>
> Key: MAPREDUCE-6801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6801.001.patch, mapreduce6801.002.patch
>
>
> TestKill.testKillJob often fails for the same reason with the following error
> message:
> {code}
> 1 tests failed.
> FAILED: org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob
> Error Message:
> Task state not correct expected: but was:
> Stack Trace:
> java.lang.AssertionError: Task state not correct expected: but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:84)
> {code}
> The root cause is that when the job is in KILLED state from an external view,
> TaskKillEvents and TaskAttemptKillEvents placed on the event loop queue may
> not have been processed by the dispatcher thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6801) Fix flaky TestKill.testKillJob()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675040#comment-15675040
]
Varun Saxena edited comment on MAPREDUCE-6801 at 11/17/16 10:49 PM:
Thanks [~haibochen] for the patch. This should handle all the cases except one,
although that would happen rarely. If internal state at which job is stuck is
SETUP (due to slow processing), tasks wont be scheduled. Hence, task wont reach
kill state for which we have an assertion for. Internal state of SETUP means an
external state of RUNNING. Therefore {{app.waitForState(job,
JobState.RUNNING)}} should be replaced by {{app.waitForInternalState((JobImpl)
job, JobStateInternal.RUNNING)}}
I was able to simulate this case by putting a sleep in dispatcher.
was (Author: varun_saxena):
Thanks [~haibochen] for the patch. This should handle all the cases except one,
although rarely. If internal state at which job is stuck is SETUP (due to slow
processing), tasks wont be scheduled. Hence, task wont reach kill state for
which we have an assertion for. Internal state of SETUP means an external state
of RUNNING. Therefore {{app.waitForState(job, JobState.RUNNING)}} should be
replaced by {{app.waitForInternalState((JobImpl) job,
JobStateInternal.RUNNING)}}
I was able to simulate this case by putting a sleep in dispatcher.
> Fix flaky TestKill.testKillJob()
>
>
> Key: MAPREDUCE-6801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6801.001.patch
>
>
> TestKill.testKillJob often fails for the same reason with the following error
> message:
> {code}
> 1 tests failed.
> FAILED: org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob
> Error Message:
> Task state not correct expected: but was:
> Stack Trace:
> java.lang.AssertionError: Task state not correct expected: but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:84)
> {code}
> The root cause is that when the job is in KILLED state from an external view,
> TaskKillEvents and TaskAttemptKillEvents placed on the event loop queue may
> not have been processed by the dispatcher thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6801) Fix flaky TestKill.testKillJob()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675040#comment-15675040
]
Varun Saxena commented on MAPREDUCE-6801:
-
Thanks [~haibochen] for the patch. This should handle all the cases except one,
although rarely. If internal state at which job is stuck is SETUP (due to slow
processing), tasks wont be scheduled. Hence, task wont reach kill state for
which we have an assertion for. Internal state of SETUP means an external state
of RUNNING. Therefore {{app.waitForState(job, JobState.RUNNING)}} should be
replaced by {{app.waitForInternalState((JobImpl) job,
JobStateInternal.RUNNING)}}
I was able to simulate this case by putting a sleep in dispatcher.
> Fix flaky TestKill.testKillJob()
>
>
> Key: MAPREDUCE-6801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6801
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: mapreduce6801.001.patch
>
>
> TestKill.testKillJob often fails for the same reason with the following error
> message:
> {code}
> 1 tests failed.
> FAILED: org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob
> Error Message:
> Task state not correct expected: but was:
> Stack Trace:
> java.lang.AssertionError: Task state not correct expected: but
> was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.mapreduce.v2.app.TestKill.testKillJob(TestKill.java:84)
> {code}
> The root cause is that when the job is in KILLED state from an external view,
> TaskKillEvents and TaskAttemptKillEvents placed on the event loop queue may
> not have been processed by the dispatcher thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6811:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~brahmareddy] for your contribution. I have committed this to trunk,
branch-2.
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670998#comment-15670998
]
Varun Saxena commented on MAPREDUCE-6811:
-
Thanks [~brahmareddy] for the patch. Changes LGTM. Will commit it after a while.
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6811:
Assignee: Brahma Reddy Battula
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6782:
Component/s: jobhistoryserver
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6782.01.patch, MAPREDUCE-6782.02.patch
>
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6782:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.9.0
Status: Resolved (was: Patch Available)
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6782.01.patch, MAPREDUCE-6782.02.patch
>
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647011#comment-15647011
]
Varun Saxena commented on MAPREDUCE-6782:
-
Committing it shortly.
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
> Attachments: MAPREDUCE-6782.01.patch, MAPREDUCE-6782.02.patch
>
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643200#comment-15643200
]
Varun Saxena commented on MAPREDUCE-6782:
-
Thanks [~ajithshetty] for the patch.
Changes LGTM. Tests are not required as this is a UI related issue.
Will commit it by tomorrow unless there are further comments.
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
> Attachments: MAPREDUCE-6782.01.patch, MAPREDUCE-6782.02.patch
>
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639340#comment-15639340
]
Varun Saxena commented on MAPREDUCE-6782:
-
Thanks [~ajithshetty] for the patch.
Changes LGTM. I tested the UI and am able to search each column in tasks page.
Checkstyle issue can be fixed though.
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
> Attachments: MAPREDUCE-6782.01.patch
>
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6541) Exclude scheduled reducer memory when calculating available mapper slots from headroom to avoid deadlock
[
https://issues.apache.org/jira/browse/MAPREDUCE-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611608#comment-15611608
]
Varun Saxena commented on MAPREDUCE-6541:
-
[~Naganarasimha], want me to fix checkstyle ? Most of them (i.e. whitespace
after { ) are false negatives
> Exclude scheduled reducer memory when calculating available mapper slots from
> headroom to avoid deadlock
> -
>
> Key: MAPREDUCE-6541
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6541
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Wangda Tan
>Assignee: Varun Saxena
> Attachments: MAPREDUCE-6541.01.patch, MAPREDUCE-6541.02.patch
>
>
> We saw a MR deadlock recently:
> - When NM restarted by framework without enable recovery, containers running
> on these nodes will be identified as "ABORTED", and MR AM will try to
> reschedule "ABORTED" mapper containers.
> - Since such lost mappers are "ABORTED" container, MR AM gives normal mapper
> priority (priority=20) to such mapper requests. If there's any pending
> reducer (priority=10) at the same time, mapper requests need to wait for
> reducer requests satisfied.
> - In our test, one mapper needs 700+ MB, reducer needs 1000+ MB, and RM
> available resource = mapper-request = (700+ MB), only one job was running in
> the system so scheduler cannot allocate more reducer containers AND MR-AM
> thinks there're enough headroom for mapper so reducer containers will not be
> preempted.
> MAPREDUCE-6302 can solve most of the problems, but in the other hand, I think
> we may need to exclude scheduled reducers resource when calculating
> #available-mapper-slots from headroom. Which we can avoid excessive reducer
> preemption.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
