[jira] [Updated] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6519: Fix Version/s: HADOOP-17800 > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Fix For: HADOOP-17800 > > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch, MAPREDUCE-6519-HADOOP-17800.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6519: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Committed to HADOOP-17800 branach. [~newanja] and [~hemanthboyina] thanks for your contribution. > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch, MAPREDUCE-6519-HADOOP-17800.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6519) Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/MAPREDUCE-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390992#comment-17390992 ] Brahma Reddy Battula commented on MAPREDUCE-6519: - [~hemanthboyina] thanks for uploading. Patch LGTM. > Avoid unsafe split and append on fields that might be IPv6 literals > --- > > Key: MAPREDUCE-6519 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6519 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic >Priority: Major > Labels: ipv6 > Attachments: MAPREDUCE-6519-HADOOP-11890.1.patch, > MAPREDUCE-6519-HADOOP-11890.2.patch, MAPREDUCE-6519-HADOOP-17800.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > mapreduce portion of patch in HADOOP-12122 that couldn't run due to number of > components touched. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7344) Improve logging by removing string Concatenation and change level of logs from INFO to WARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346398#comment-17346398 ] Brahma Reddy Battula commented on MAPREDUCE-7344: - [~dmmkr] How about splitting this task to module wise..? > Improve logging by removing string Concatenation and change level of logs > from INFO to WARN > --- > > Key: MAPREDUCE-7344 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7344 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > Attachments: MAPREDUCE-7344.001.patch, MAPREDUCE-7344.002.patch > > > 1. Currently in few of debug logs string Concatenation is performed without > any debug check, this can be fixed by passing the variables as arguments. > 2. Few logs while catching exception can be moved from INFO to WARN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7199) HsJobsBlock reuse JobACLsManager for checkAccess
[
https://issues.apache.org/jira/browse/MAPREDUCE-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-7199:
Fix Version/s: 3.3.1
> HsJobsBlock reuse JobACLsManager for checkAccess
>
>
> Key: MAPREDUCE-7199
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7199
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-7199-001.patch, MAPREDUCE-7199.002.patch,
> MAPREDUCE-7199.003.patch
>
>
> Reuse JobAclManager.checkAccess
> {code}
> private boolean checkAccess(String userName) {
> if(!areAclsEnabled) {
> return true;
> }
> // User could see its own job.
> if (ugi.getShortUserName().equals(userName)) {
> return true;
> }
> // Admin could also see all jobs
> if (adminAclList != null && adminAclList.isUserAllowed(ugi)) {
> return true;
> }
> return false;
> }
> {code}
> {code}
> jobACLsManager
> .checkAccess(ugi, JobACL.VIEW_JOB, ..
> new AccessControlList()))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7199) HsJobsBlock reuse JobACLsManager for checkAccess
[
https://issues.apache.org/jira/browse/MAPREDUCE-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313941#comment-17313941
]
Brahma Reddy Battula commented on MAPREDUCE-7199:
-
Done.Committed to branch-3.3..
> HsJobsBlock reuse JobACLsManager for checkAccess
>
>
> Key: MAPREDUCE-7199
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7199
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: MAPREDUCE-7199-001.patch, MAPREDUCE-7199.002.patch,
> MAPREDUCE-7199.003.patch
>
>
> Reuse JobAclManager.checkAccess
> {code}
> private boolean checkAccess(String userName) {
> if(!areAclsEnabled) {
> return true;
> }
> // User could see its own job.
> if (ugi.getShortUserName().equals(userName)) {
> return true;
> }
> // Admin could also see all jobs
> if (adminAclList != null && adminAclList.isUserAllowed(ugi)) {
> return true;
> }
> return false;
> }
> {code}
> {code}
> jobACLsManager
> .checkAccess(ugi, JobACL.VIEW_JOB, ..
> new AccessControlList()))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312508#comment-17312508
]
Brahma Reddy Battula commented on MAPREDUCE-6826:
-
Done, Cherry-picked to branch-3.3.
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
>
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-6826-001.patch, MAPREDUCE-6826-002.patch,
> MAPREDUCE-6826-003.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
>
[jira] [Updated] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
[
https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6826:
Fix Version/s: 3.3.1
> Job fails with InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING
>
>
> Key: MAPREDUCE-6826
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Varun Saxena
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: MAPREDUCE-6826-001.patch, MAPREDUCE-6826-002.patch,
> MAPREDUCE-6826-003.patch
>
>
> This happens if a container is preempted by scheduler after job starts
> committing.
> And this exception in turn leads to application being marked as FAILED in
> YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state
> is COMMITTING or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if
> job is already in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1482404625971_23910_m_04 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e55_1482404625971_23910_01_10 taskAttempt
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1482404625971_23910_m_04_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for
> JobFinishedEvent
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly,
> recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator
> isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified
> that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry:
> true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
> JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the
> services
> 2016-12-23 09:10:38,800 INFO [Thread-93]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0
> HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311099#comment-17311099 ] Brahma Reddy Battula commented on MAPREDUCE-6749: - [~devaraj] there are conflicts while rebasing against the trunk as this branch was old. Hence i am thinking create a new branch for this,please let me know your thoughts on this..? > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6726) YARN Registry based AM discovery with retry and in-flight task persistent via JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290657#comment-17290657 ] Brahma Reddy Battula commented on MAPREDUCE-6726: - [~junping_du]/[~srikanth.sampath]/[~jianhe] could you please review this and move forward, this is asked for 3.3.1 release.. > YARN Registry based AM discovery with retry and in-flight task persistent via > JHS > - > > Key: MAPREDUCE-6726 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6726 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Reporter: Junping Du >Assignee: D M Murali Krishna Reddy >Priority: Major > Fix For: 3.2.2 > > Attachments: MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.001.patch, > MAPREDUCE-6726-MAPREDUCE-6608.002.patch, > MAPREDUCE-6726-MAPREDUCE-6608.003.patch, MAPREDUCE-6726.004.patch, > MAPREDUCE-6726.005.trunk.patch, TestReport.pdf, WorkPreservingMRAppMaster.pdf > > > Several tasks will be achieved in this JIRA based on the demo patch in > MAPREDUCE-6608: > 1. AM discovery base on YARN register service. Could be replaced by YARN-4758 > later due to scale up issue. > 2. Retry logic for TaskUmbilicalProtocol RPC connection > 3. In-flight task recover after AM restart via JHS > 4. Configuration to control the behavior compatible with previous when not > enable this feature (by default). > All security related issues and other concerns discussed in MAPREDUCE-6608 > will be addressed in follow up JIRAs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288837#comment-17288837 ] Brahma Reddy Battula commented on MAPREDUCE-6749: - [~BilwaST] thanks for attaching the performance report. Looks performance is better when we enable this feaute.. It can be more benefit when more number of container's are there.. you can give how much performabce gain with this also. [~devaraj] can we merge pending jira's and merge this feature..THinking to have for 3.3.1 relase based on stablity. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288829#comment-17288829 ] Brahma Reddy Battula commented on MAPREDUCE-7169: - [~ahussein] , [~lichen1109] and [~epayne] [~BilwaST] this Jira marked for 3.3.1 release which is going to release near future, please let me know update on this. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200059#comment-17200059 ] Brahma Reddy Battula commented on MAPREDUCE-6749: - [~devaraj] looks no progress on this, we would like know the status and work on this. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7119) Avoid stopContainer on failed node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7119: Target Version/s: 3.4.0 (was: 3.3.0) > Avoid stopContainer on failed node > -- > > Key: MAPREDUCE-7119 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7119 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.8.4 >Reporter: Brandon Scheller >Priority: Minor > Labels: bug, bugfix, stopContainer > > Avoid stopContainer on failed node > > If a container failed to launch earlier due to terminated instances, it has > already been removed from the container hash map. Avoiding the kill() for > CONTAINER_REMOTE_CLEANUP will avoid wasting 15min per container on > retries/timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6851) Terasort does not work with S3 FileSystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6851: Target Version/s: 3.4.0 (was: 3.3.0) > Terasort does not work with S3 FileSystem > - > > Key: MAPREDUCE-6851 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6851 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Priority: Major > > Terasort currently writes a partition list file in the output directory and > add it to distributed cache. If a S3 bucket is used as the output directory, > the job may fail because the partition list file will be localized by NMs > which may not have the s3 credentials. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7227: Target Version/s: 3.4.0 (was: 3.3.0, 3.1.4) > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch, > Process_Analysis.png > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7070) Batch calls to RackResolver from AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7070: Target Version/s: 3.4.0 (was: 3.3.0) > Batch calls to RackResolver from AM > --- > > Key: MAPREDUCE-7070 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7070 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, performance >Reporter: Greg Phillips >Priority: Minor > Attachments: MAPREDUCE-7070.patch > > > TaskAttemptImpl calls RackResolver once for every split location. This > results in many shells being spawned by the Application Master when the > topology script is enabled. The worst case scenario would entail one call to > the topology script for each Datanode. The NameNode is also topology aware, > and uses batch calls to the topology script to improve performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6996) FileInputFormat#getBlockIndex should include file name in the exception.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6996:
Target Version/s: 3.4.0 (was: 3.3.0)
> FileInputFormat#getBlockIndex should include file name in the exception.
>
>
> Key: MAPREDUCE-6996
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6996
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh Shah
>Priority: Minor
> Labels: newbie++
>
> {code:title=FileInputFormat..java|borderStyle=solid}
> // Some comments here
> protected int getBlockIndex(BlockLocation[] blkLocations,
> long offset) {
> {
> ...
> ...
> BlockLocation last = blkLocations[blkLocations.length -1];
> long fileLength = last.getOffset() + last.getLength() -1;
> throw new IllegalArgumentException("Offset " + offset +
>" is outside of file (0.." +
>fileLength + ")");
> }
> {code}
> When the file is open for writing, the {{last.getLength()}} and
> {{last.getOffset()}} will be zero and we see the following exception stack
> trace.
> {noformat}
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:288)
> Caused by: java.lang.IllegalArgumentException: Offset 0 is outside of file
> (0..-1)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getBlockIndex(FileInputFormat.java:453)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:413)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:265)
> ... 18 more
> {noformat}
> Its difficult to debug which file was open.
> So creating this ticket to include the filename in the exception.
> Since {{FileInputFormat#getBlockIndex}} is protected, we can't change the
> signature of that method and add file name to arguments.
> The only way I can think to fix this is:
> {code:title=FileInputFormat..java|borderStyle=solid}
> public InputSplit[] getSplits(JobConf job, int numSplits)
> throws IOException {
> {
> ...
> ...
>for (FileStatus file: files) {
> Path path = file.getPath();
> long length = file.getLen();
> if (length != 0) {
> FileSystem fs = path.getFileSystem(job);
> BlockLocation[] blkLocations;
> if (file instanceof LocatedFileStatus) {
> blkLocations = ((LocatedFileStatus) file).getBlockLocations();
> } else {
> blkLocations = fs.getFileBlockLocations(file, 0, length);
> }
> if (isSplitable(fs, path)) {
> long blockSize = file.getBlockSize();
> long splitSize = computeSplitSize(goalSize, minSize, blockSize);
> long bytesRemaining = length;
> while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
> String[][] splitHosts = getSplitHostsAndCachedHosts(blkLocations,
> length-bytesRemaining, splitSize, clusterMap);
> splits.add(makeSplit(path, length-bytesRemaining, splitSize,
> splitHosts[0], splitHosts[1]));
> bytesRemaining -= splitSize;
> }
> if (bytesRemaining != 0) {
> String[][] splitHosts = getSplitHostsAndCachedHosts(blkLocations,
> length
> - bytesRemaining, bytesRemaining, clusterMap);
> splits.add(makeSplit(path, length - bytesRemaining,
> bytesRemaining,
> splitHosts[0], splitHosts[1]));
> }
> } else {
> String[][] splitHosts =
> getSplitHostsAndCachedHosts(blkLocations,0,length,clusterMap);
> splits.add(makeSplit(path, 0, length, splitHosts[0],
> splitHosts[1]));
> }
> } else {
> //Create empty hosts array for zero length files
> splits.add(makeSplit(path, 0, length, new String[0]));
> }
> }
> {code}
> Have a try-catch block around the above code chunk and catch
> {{IllegalArgumentException}} and check for message {{Offset 0 is outside of
> file (0..-1)}}.
> If yes, add the file name and rethrow {{IllegalArgumentException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6467) Submitting streaming job is not thread safe
[ https://issues.apache.org/jira/browse/MAPREDUCE-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6467: Target Version/s: 3.4.0 (was: 3.3.0) > Submitting streaming job is not thread safe > --- > > Key: MAPREDUCE-6467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 2.7.1 >Reporter: jeremie simon >Assignee: Ivo Udelsmann >Priority: Minor > Labels: easyfix, streaming, thread-safety > Attachments: MAPREDUCE-6467.001.patch > > > The submission of the streaming job is not thread safe. > That is because the class StreamJob is using the OptionBuilder which is > itself not thread safe. > This can cause super tricky bugs. > An easy fix would be to simply create instances of Option through the normal > constructor and decorate the object if necessary. > This fix should be applied on the functions createOption and > createBoolOption. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7185) Parallelize part files move in FileOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-7185:
Target Version/s: 3.4.0 (was: 3.3.0, 2.10.1)
> Parallelize part files move in FileOutputCommitter
> --
>
> Key: MAPREDUCE-7185
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7185
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 3.2.0, 2.9.2
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Attachments: MAPREDUCE-7185.patch
>
>
> If map task outputs multiple files it could be slow to move them from temp
> directory to output directory in object stores (GCS, S3, etc).
> To improve performance we need to parallelize move of more than 1 file in
> FileOutputCommitter.
> Repro:
> Start spark-shell:
> {code}
> spark-shell --num-executors 2 --executor-memory 10G --executor-cores 4 --conf
> spark.dynamicAllocation.maxExecutors=2
> {code}
> From spark-shell:
> {code}
> val df = (1 to 1).toList.toDF("value").withColumn("p", $"value" %
> 10).repartition(50)
> df.write.partitionBy("p").mode("overwrite").format("parquet").options(Map("path"
> -> s"gs://some/path")).saveAsTable("parquet_partitioned_bench")
> {code}
> With the fix execution time reduces from 130 seconds to 50 seconds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7236) HadoopArchiveLogs will use token to create proxy user when kerberos on
[ https://issues.apache.org/jira/browse/MAPREDUCE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7236: Target Version/s: 3.4.0 (was: 3.3.0, 2.9.3) > HadoopArchiveLogs will use token to create proxy user when kerberos on > -- > > Key: MAPREDUCE-7236 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7236 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.9.2 >Reporter: Yicong Cai >Priority: Major > > HadoopArchiveLogsRunner runs in the Yarn Container via DistributedShell. > The Client of the DistributedShell gets the Token and uses it for the Runner. > The Runner create ProxyUser via Token, which violates the ProxyUser principle. > There are two solutions: > 1. Pass the Keytab to the Runner, login with Keytab and create ProxyUser. > 2. Run the HadoopArchiveLogs task with HDFS Super User. After the Archive is > finished, use chown to modify it to the corresponding user. > I prefer to use the first way to solve the problem. Any suggestions? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7120) Make hadoop consider wildcard host as datalocal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7120: Target Version/s: 3.4.0 (was: 3.3.0) > Make hadoop consider wildcard host as datalocal > --- > > Key: MAPREDUCE-7120 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7120 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Affects Versions: 2.8.4 >Reporter: Brandon Scheller >Priority: Major > Labels: feature, option, performance > > Make hadoop consider wildcard host as datalocal. > This allows hadoop to treat "\*" as data local. This allows remote > filesystems to increase performance by skipping retrys for data locality by > returning "\*" as the block host when the filesystem is asked for the block > location. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6946) Moving logging APIs over to slf4j in hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6946: Target Version/s: 3.4.0 (was: 3.3.0) > Moving logging APIs over to slf4j in hadoop-mapreduce > - > > Key: MAPREDUCE-6946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6946 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira Ajisaka >Priority: Major > > MapReduce side of YARN-6712. This is an umbrella jira for MapReduce. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6854) Each map task should create a unique temporary name that includes an object name
[ https://issues.apache.org/jira/browse/MAPREDUCE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6854: Target Version/s: 3.4.0 (was: 3.3.0) > Each map task should create a unique temporary name that includes an object > name > > > Key: MAPREDUCE-6854 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6854 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 >Reporter: Gil Vernik >Priority: Major > Labels: patch > Attachments: HADOOP-6854-001.patch, HADOOP-6854-002.patch > > > Consider an example: a local file "/data/a.txt" need to be copied into > swift://container.service/data/a.txt > The way distcp works is that first it will upload "/data/a.txt" into > swift://container.mil01/data/.distcp.tmp.attempt_local2036034928_0001_m_00_0 > Upon completion distcp will move > swift://container.mil01/data/.distcp.tmp.attempt_local2036034928_0001_m_00_0 > into swift://container.mil01/data/a.txt > > The temporary file naming convention assumes that each map task will > sequentially create objects as swift://container.mil01/.distcp.tmp.attempt_ID > and then rename them to the final names. Most of Hadoop eco system > components use object.name which is part of the temporary name, however > distcp doesn't use such approach. > This JIRA propose to add a configuration key indicating that temporary > objects will also include object name as part of their temporary file name, > For example > "/data/a.txt" will be uploaded into > "swift://container.mil01/data/a.txt.distcp.tmp.attempt_local2036034928_0001_m_00_0" > "a.txt.distcp.tmp.attempt_local2036034928_0001_m_00_0" doesn't affects > flows in the access drivers, since "a.txt" is not considered as a > sub-directory so no special operations will be taken. > The benefits of the patch : > 1. Temp object names will be better distributed in object stores, since they > all has different prefix. > 2. Sometimes it's not possible to debug what data is copied and what failed. > Sometimes temp files are not renamed, it will be much better if expecting > temp name - one can figure what object names were copied. > 3. Different systems may expect > "a.txt.distcp.tmp.attempt_local2036034928_0001_m_00_0" and extract value > prior "distcp.tmp" thus getting destination object name. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6458) Figure out the way to pass build-in classpath (files in distributed cache, etc.) from parent to spawned shells
[ https://issues.apache.org/jira/browse/MAPREDUCE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6458: Target Version/s: 3.4.0 (was: 3.3.0) > Figure out the way to pass build-in classpath (files in distributed cache, > etc.) from parent to spawned shells > -- > > Key: MAPREDUCE-6458 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6458 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Junping Du >Assignee: Dustin Cote >Priority: Major > Attachments: MAPREDUCE-6458.00.patch > > > In MAPREDUCE-6454 (target for branch-2.x), we provide a way with constraints > to pass built-in classpath from parent to child shell, via HADOOP_CLASSPATH, > so jars in distributed cache can still work in child tasks. In trunk, we may > think some way different, like: involve additional env var to safely pass > build-in classpath. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6901) Remove @deprecated tags from DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6901: Target Version/s: 3.4.0 (was: 3.3.0) > Remove @deprecated tags from DistributedCache > - > > Key: MAPREDUCE-6901 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6901 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: distributed-cache >Affects Versions: 3.0.0-alpha3 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Major > Attachments: MAPREDUCE-6901.001.patch > > > Doing this as part of Hadoop 3 cleanup. > DistributedCache has been marked as deprecated forever to the point where the > change that did it isn't in Git. > I don't really have a preference for whether we remove it or not, but I'd > like to have a discussion and have it properly documented as a release not > for Hadoop 3 before we hit final release. At the very least we can have a > Release Note that will sum up whatever discussion we have here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6857) Reduce number of exists() calls on the target object
[ https://issues.apache.org/jira/browse/MAPREDUCE-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6857: Target Version/s: 3.4.0 (was: 3.3.0) > Reduce number of exists() calls on the target object > > > Key: MAPREDUCE-6857 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6857 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 >Reporter: Gil Vernik >Priority: Major > Attachments: HADOOP-6857-002.patch > > > CopyMapper.map(..) calls targetStatus = targetFS.getFileStatus(target). > Few steps later RetriableFileCopyCommand.promoteTmpToTarget(..) will call > again exists(target) and delete if present. > The second exists() is useless, since if targetStatus==null it can be easily > seen if overwrite mode is activated and so target object can be deleted. > The propose of this patch is to delete target object by using targetStatus > and thus avoid calling exists() method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7182) MapReduce input format/record readers to support S3 select queries
[ https://issues.apache.org/jira/browse/MAPREDUCE-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7182: Target Version/s: 3.4.0 (was: 3.3.0) > MapReduce input format/record readers to support S3 select queries > -- > > Key: MAPREDUCE-7182 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7182 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Major > > HADOOP-15229 adds S3 select through the (new) async openFile API, but the > classic RecordReader can't handle it because > # the files are shorter than they are in a getFileStatus, and the readers > assume that an EOFException is an error in that situation > # everything assumes plain text is splitable > # if a file has a gz extension, the gunzip codec should be used. So breaks > transcoded/uncompressed data > to handle s3 select data sources we need to be able to address them, either > through changes to the existing code (danger?) or some new readers -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7235) NPE in HadoopArchiveLogs#filterAppsByAggregatedStatus
[
https://issues.apache.org/jira/browse/MAPREDUCE-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-7235:
Target Version/s: 3.4.0 (was: 3.3.0, 2.9.3)
> NPE in HadoopArchiveLogs#filterAppsByAggregatedStatus
> -
>
> Key: MAPREDUCE-7235
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7235
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Yicong Cai
>Priority: Major
> Attachments: MAPREDUCE-7235.01.patch, MAPREDUCE-7235.branch-2.01.patch
>
>
> If the user running the Archive Logs task does not have the VIEW_APP
> permission of an APP, LogAggregationStatus is Null, and
> HadoopArchiveLogs#filterAppsByAggregatedStatus does not perform NPE check.
>
> {panel:title=NPE exception}
> NullPointerExceptionNullPointerExceptionjava.lang.NullPointerException at
> org.apache.hadoop.tools.HadoopArchiveLogs.filterAppsByAggregatedStatus(HadoopArchiveLogs.java:306)
> at org.apache.hadoop.tools.HadoopArchiveLogs.run(HadoopArchiveLogs.java:157)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at
> org.apache.hadoop.tools.HadoopArchiveLogs.main(HadoopArchiveLogs.java:119)
> {panel}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6903) NPE occured when run MapredTestDriver's testcase
[
https://issues.apache.org/jira/browse/MAPREDUCE-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6903:
Target Version/s: 3.4.0 (was: 3.3.0)
> NPE occured when run MapredTestDriver's testcase
>
>
> Key: MAPREDUCE-6903
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6903
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Labels: easyfix
> Attachments: MAPREDUCE-6903.001.patch, MAPREDUCE-6903.002.patch,
> MAPREDUCE-6903.003.patch, MAPREDUCE-6903.004.patch
>
>
> When runing the timelineperformance test from MapredTestDriver without input
> parameter, an NPE was throwed:
> {quote}
> ./hadoop org.apache.hadoop.test.MapredTestDriver timelineperformance
> ...
> 2017-06-08 09:32:13,194 ERROR mapreduce.SimpleEntityWriterV1: writing to the
> timeline service failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:178)
> at
> org.apache.hadoop.mapreduce.SimpleEntityWriterV1.map(SimpleEntityWriterV1.java:82)
> at
> org.apache.hadoop.mapreduce.SimpleEntityWriterV1.map(SimpleEntityWriterV1.java:41)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> That's better to add a protection of NULL pointer check.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6734) Add option to distcp to preserve file path structure of source files at the destination
[ https://issues.apache.org/jira/browse/MAPREDUCE-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6734: Target Version/s: 3.4.0 (was: 3.3.0) > Add option to distcp to preserve file path structure of source files at the > destination > --- > > Key: MAPREDUCE-6734 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6734 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distcp >Affects Versions: 3.0.0-alpha2 > Environment: Software platform >Reporter: Frederick Tucker >Priority: Major > Labels: distcp, newbie, patch > Attachments: MAPREDUCE-6734.3.0.0-alpha2.patch, > MAPREDUCE-6734.3.0.0-alpha2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When copying files using distcp with globbed source files, all the matched > files in the glob are copied in a single flat directory. This causes > problems when the file structure at the source is important. It also is an > issue when there are two files matched in the glob with the same name because > it causes a duplicate file error at the target. I'd like to have an option > to preserve the file structure of the source files when globbing inputs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7002) Shows processed record count with _SUCCESS file
[ https://issues.apache.org/jira/browse/MAPREDUCE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7002: Target Version/s: 3.4.0 (was: 3.3.0) > Shows processed record count with _SUCCESS file > --- > > Key: MAPREDUCE-7002 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7002 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 3.0.0-beta1 >Reporter: Mehul Garnara (MG) >Assignee: Mehul Garnara (MG) >Priority: Minor > > I'm using spark jobs for processing files and found requirement for getting > number of records process after spark submit ran successfully. After analyze > it we realized spark using hadoop underline where hadoop is generating the > _SUCCESS file as an indicator of the successful completion. I would like to > propose requirement for adding number of count with the file name, for > example for 100 record processed it would generate like "100_SUCCESS" this > will give ability to know how many records get processed on completion of > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7155) TestHSAdminServer is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7155: Target Version/s: 3.4.0 (was: 3.3.0) > TestHSAdminServer is failing > > > Key: MAPREDUCE-7155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7155 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Jason Darrell Lowe >Assignee: David Mollitor >Priority: Major > Attachments: MAPREDUCE-7155.1.patch > > > After HADOOP-15836 TestHSAdminServer has been failing consistently. Sample > stacktraces to follow. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6241: Target Version/s: 3.4.0 (was: 3.3.0) > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 2.6.0, 3.0.0-alpha1 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Assignee: Binglin Chang >Priority: Major > Labels: BB2015-05-TBR, features > Attachments: MAPREDUCE-6241.001.patch, MAPREDUCE-6241.002.patch, > MAPREDUCE-6241.003.patch > > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6925) CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6925:
Target Version/s: 3.4.0 (was: 3.3.0)
> CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and
> YarnChild
> ---
>
> Key: MAPREDUCE-6925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6925) CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild
[
https://issues.apache.org/jira/browse/MAPREDUCE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079625#comment-17079625
]
Brahma Reddy Battula commented on MAPREDUCE-6925:
-
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> CLONE - Make Counter limits consistent across JobClient, MRAppMaster, and
> YarnChild
> ---
>
> Key: MAPREDUCE-6925
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6925
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, client, task
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
>
> Currently, counter limits "mapreduce.job.counters.*" handled by
> {{org.apache.hadoop.mapreduce.counters.Limits}} are initialized
> asymmetrically: on the client side, and on the AM, job.xml is ignored whereas
> it's taken into account in YarnChild.
> It would be good to make the Limits job-configurable, such that max
> counters/groups is only increased when needed. With the current Limits
> implementation relying on static constants, it's going to be challenging for
> tools that submit jobs concurrently without resorting to class loading
> isolation.
> The patch that I am uploading is not perfect but demonstrates the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7264) overall reduction of ApplicationMaster exit because of unhandled TA_TOO_MANY_FETCH_FAILURE event
[ https://issues.apache.org/jira/browse/MAPREDUCE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079587#comment-17079587 ] Brahma Reddy Battula commented on MAPREDUCE-7264: - Removed the fix version , as this issue is not resloved.. [~tuyu] please update the targetversion for this issue. > overall reduction of ApplicationMaster exit because of unhandled > TA_TOO_MANY_FETCH_FAILURE event > > > Key: MAPREDUCE-7264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 3.2.1 >Reporter: tuyu >Priority: Critical > Attachments: MAPREDUCE-7264-branch-3.2.001.patch > > > when rolling restart nodemanager, some mapreduce job will exit because of > unhandle TA_TOO_MANY_FETCH_FAILURE event > details: >if task stay in SUCCEEDED state, now reciveice TA_TOO_MANY_FETCH_FAILURE > event,AM will handle this situation correct,but if stay in > SUCCESS_FINISHING_CONTAINER or some other state,will exit by invalid event > [YARN-1469|https://issues.apache.org/jira/browse/YARN-1469] > [MAPREDUCE-7240|https://issues.apache.org/jira/browse/MAPREDUCE-7240] > [MAPREDUCE-7249|https://issues.apache.org/jira/browse/MAPREDUCE-7249] > [MAPREDUCE-7240|https://issues.apache.org/jira/browse/MAPREDUCE-5409] >reason: > when map task send done rpc to AM, AM will Transition this task to > SUCCESS_FINISHING_CONTAINER state, and add this task to > mapAttemptCompletionEvents List, when reduce send > getMapAttemptCompletionEvents rpc to get the complete map, the task stay in > SUCCESS_FINISHING_CONTAINER state will return. but if now,NM is restart or > stop,many reducer task will shuffle fail,and report to AM, AM will send > TA_TOO_MANY_FETCH_FAILURE event,if map task state cannot handle > TA_TOO_MANY_FETCH_FAILURE event,AM will exit. > i found isusses to resolve this problem,but not cover all situation. > The state Transition from SUCCESS_FINISHING_CONTAINER will reciveice > TA_TOO_MANY_FETCH_FAILURE event，like > (SUCCEEDED,SUCCESS_CONTAINER_CLEANUP,SUCCESS_FINISHING_CONTAINER,FAILED,KILL_CONTAINER_CLEANUP) > In hadoop 3.2.1, only SUCCEEDED,FAILED AND KILLED state can handle > TA_TOO_MANY_FETCH_FAILURE event, and some jira to fix > SUCCESS_CONTAINER_CLEANUP,SUCCESS_FINISHING_CONTAINER,KILLED,but > KILL_CONTAINER_CLEANUP,KILL_TASK_CLEANUP also should to handle > TA_TOO_MANY_FETCH_FAILURE event -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7264) overall reduction of ApplicationMaster exit because of unhandled TA_TOO_MANY_FETCH_FAILURE event
[ https://issues.apache.org/jira/browse/MAPREDUCE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-7264: Fix Version/s: (was: 3.2.2) (was: 3.1.4) (was: 3.3.0) > overall reduction of ApplicationMaster exit because of unhandled > TA_TOO_MANY_FETCH_FAILURE event > > > Key: MAPREDUCE-7264 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7264 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 3.2.1 >Reporter: tuyu >Priority: Critical > Attachments: MAPREDUCE-7264-branch-3.2.001.patch > > > when rolling restart nodemanager, some mapreduce job will exit because of > unhandle TA_TOO_MANY_FETCH_FAILURE event > details: >if task stay in SUCCEEDED state, now reciveice TA_TOO_MANY_FETCH_FAILURE > event,AM will handle this situation correct,but if stay in > SUCCESS_FINISHING_CONTAINER or some other state,will exit by invalid event > [YARN-1469|https://issues.apache.org/jira/browse/YARN-1469] > [MAPREDUCE-7240|https://issues.apache.org/jira/browse/MAPREDUCE-7240] > [MAPREDUCE-7249|https://issues.apache.org/jira/browse/MAPREDUCE-7249] > [MAPREDUCE-7240|https://issues.apache.org/jira/browse/MAPREDUCE-5409] >reason: > when map task send done rpc to AM, AM will Transition this task to > SUCCESS_FINISHING_CONTAINER state, and add this task to > mapAttemptCompletionEvents List, when reduce send > getMapAttemptCompletionEvents rpc to get the complete map, the task stay in > SUCCESS_FINISHING_CONTAINER state will return. but if now,NM is restart or > stop,many reducer task will shuffle fail,and report to AM, AM will send > TA_TOO_MANY_FETCH_FAILURE event,if map task state cannot handle > TA_TOO_MANY_FETCH_FAILURE event,AM will exit. > i found isusses to resolve this problem,but not cover all situation. > The state Transition from SUCCESS_FINISHING_CONTAINER will reciveice > TA_TOO_MANY_FETCH_FAILURE event，like > (SUCCEEDED,SUCCESS_CONTAINER_CLEANUP,SUCCESS_FINISHING_CONTAINER,FAILED,KILL_CONTAINER_CLEANUP) > In hadoop 3.2.1, only SUCCEEDED,FAILED AND KILLED state can handle > TA_TOO_MANY_FETCH_FAILURE event, and some jira to fix > SUCCESS_CONTAINER_CLEANUP,SUCCESS_FINISHING_CONTAINER,KILLED,but > KILL_CONTAINER_CLEANUP,KILL_TASK_CLEANUP also should to handle > TA_TOO_MANY_FETCH_FAILURE event -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405770#comment-16405770
]
Brahma Reddy Battula edited comment on MAPREDUCE-6867 at 3/20/18 3:52 AM:
--
Looks YARN-679 covered this one part of this issue.
{quote}{{LOG.fatal("Halt called", ee)}} possibility of one more OOM exception
{quote}
Yes, as you also mentioned, it's logged only for test.so I feel, might not
require here.
{quote}i think option of {{XX:OnOutOfMemoryError=\"kill -9}} is dependent on
the OS. so not sure its the right approach and not aware any option is there to
set based on OS type. given that we have fix in {{ExitUtil}} is it required to
have this modification too?
{quote}
{{ExitUtil}} just can through the error, but JVM will not be killed right..?
Maybe we can add based on OS..?
was (Author: brahmareddy):
Looks YARN-679 covered this one part of this issue.
bq.{{LOG.fatal("Halt called", ee)}} possibility of one more OOM exception
Yes, as you also mentioned, it's logged only for test.
bq.i think option of {{XX:OnOutOfMemoryError=\"kill -9}} is dependent on the
OS. so not sure its the right approach and not aware any option is there to set
based on OS type. given that we have fix in {{ExitUtil}} is it required to have
this modification too?
{{ExitUtil}} just can through the error, but JVM will not be killed right..?
Maybe we can add based on OS..?
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405770#comment-16405770
]
Brahma Reddy Battula commented on MAPREDUCE-6867:
-
Looks YARN-679 covered this one part of this issue.
bq.{{LOG.fatal("Halt called", ee)}} possibility of one more OOM exception
Yes, as you also mentioned, it's logged only for test.
bq.i think option of {{XX:OnOutOfMemoryError=\"kill -9}} is dependent on the
OS. so not sure its the right approach and not aware any option is there to set
based on OS type. given that we have fix in {{ExitUtil}} is it required to have
this modification too?
{{ExitUtil}} just can through the error, but JVM will not be killed right..?
Maybe we can add based on OS..?
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6676) NNBench should Throw IOException when rename and delete fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6676:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
3.0.0-alpha4
2.9.0
Status: Resolved (was: Patch Available)
Committed to {{trunk}},{{branch-2}} and {{branch-2.8}}. [~ajisakaa] thanks a
lot for your review.
> NNBench should Throw IOException when rename and delete fails
> -
>
> Key: MAPREDUCE-6676
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2
>
> Attachments: MAPREDUCE-6676.patch
>
>
> Throw IOException when rename,delete fails, currently it's unknown to user
> when rename and delte fails..
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6676) NNBench should Throw IOException when rename and delete fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6676: Summary: NNBench should Throw IOException when rename and delete fails (was: [ NNBench ] Throw IOException when rename,delete fails) > NNBench should Throw IOException when rename and delete fails > - > > Key: MAPREDUCE-6676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE-6676.patch > > > Throw IOException when rename,delete fails, currently it's unknown to user > when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6676) [ NNBench ] Throw IOException when rename,delete fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040071#comment-16040071 ] Brahma Reddy Battula commented on MAPREDUCE-6676: - [~ajisakaa] if you chance, can you review this one..? > [ NNBench ] Throw IOException when rename,delete fails > -- > > Key: MAPREDUCE-6676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE-6676.patch > > > Throw IOException when rename,delete fails, currently it's unknown to user > when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6867:
Status: Patch Available (was: Open)
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
> Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error
[
https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948286#comment-15948286
]
Brahma Reddy Battula commented on MAPREDUCE-6867:
-
[~BilwaST] thanks for reporting this issue..
I too feel, we can have try/catch as it's not handled and there are chances OOM
can occur(simple for loop with this log statement and submit mr job with less
memory) and AM can hang forever.
[~revans2]/[~tgraves]/[~raviprak]/[~vinodkv] any thoughts on this, as you all
pitched in MAPREDUCE-4300.. thanks
> ApplicationMaster hung on OOM Error
> ---
>
> Key: MAPREDUCE-6867
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call
> {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not
> handled.
> We came across a scenario where in when we submit mapreduce application ,OOM
> error occured in {{committerEventProcessor}} and then AM did not halt and did
> not log the following.Finally AM got hang since it's not thrown to main
> thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
> *org.apache.hadoop.util.ExitUtil.halt(int, String)*
> {code}
> public static void halt(int status, String msg) throws HaltException {
> LOG.info("Halt with status " + status + " Message: " + msg);
> if (systemHaltDisabled) {
> HaltException ee = new HaltException(status, msg);
> LOG.fatal("Halt called", ee);
> if (null == firstHaltException) {
> firstHaltException = ee;
> }
> throw ee;
> }
> Runtime.getRuntime().halt(status);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800940#comment-15800940 ] Brahma Reddy Battula commented on MAPREDUCE-6662: - Thanks [~ajisakaa] > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6662) Clear ASF Warnings on test data files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15789378#comment-15789378 ] Brahma Reddy Battula commented on MAPREDUCE-6662: - any plans to merge to branch-2.8..? > Clear ASF Warnings on test data files > - > > Key: MAPREDUCE-6662 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6662 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6662.01.patch, MAPREDUCE-6662.02.patch, > MAPREDUCE-6662.03.patch, MAPREDUCE-6662.04.patch, MAPREDUCE-6662.05.patch > > > In all MAPREDUCE QA runs, '14 ASF Warnings' will mark the entire QA result as > '-1 Overall'. > These warnings are from the files generated in tests. > It will be good to see green +1s in QA report than RED -1(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672384#comment-15672384
]
Brahma Reddy Battula commented on MAPREDUCE-6811:
-
[~varun_saxena] thanks for review and commit.
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6811:
Attachment: MAPREDUCE-6811.patch
Uploaded patch..Formatted the lines to avoid the check-style issues.
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
[
https://issues.apache.org/jira/browse/MAPREDUCE-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6811:
Status: Patch Available (was: Open)
> TestPipeApplication#testSubmitter fails after HADOOP-13802
> --
>
> Key: MAPREDUCE-6811
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
> Attachments: MAPREDUCE-6811.patch
>
>
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
> {noformat}
> *Reference*
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6811) TestPipeApplication#testSubmitter fails after HADOOP-13802
Brahma Reddy Battula created MAPREDUCE-6811:
---
Summary: TestPipeApplication#testSubmitter fails after HADOOP-13802
Key: MAPREDUCE-6811
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6811
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
{noformat}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.mapred.pipes.TestPipeApplication.testSubmitter(TestPipeApplication.java:302)
{noformat}
*Reference*
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/226/
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6811/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639231#comment-15639231
]
Brahma Reddy Battula edited comment on MAPREDUCE-6782 at 11/5/16 10:35 AM:
---
[~ajithshetty] thanks for interest.
bq.I have solution for this, please assign this issue to me (already discussed
with gu-chi)
Done, Added you in contributor list and assigned to to you.
was (Author: brahmareddy):
[~ajithshetty] thanks for interest.
bq.I have solution for this, please assign this issue to me (already discussed
with gu-chi)
Done, Added you as contributor list and assigned to to you..
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15639231#comment-15639231
]
Brahma Reddy Battula commented on MAPREDUCE-6782:
-
[~ajithshetty] thanks for interest.
bq.I have solution for this, please assign this issue to me (already discussed
with gu-chi)
Done, Added you as contributor list and assigned to to you..
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6782:
Assignee: Ajith S (was: Brahma Reddy Battula)
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Ajith S
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula reassigned MAPREDUCE-6782:
---
Assignee: Brahma Reddy Battula (was: gu-chi)
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Brahma Reddy Battula
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6783) Default setting 'inheritParentEnv=false' caused mr executor failed to initialize
[ https://issues.apache.org/jira/browse/MAPREDUCE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529432#comment-15529432 ] Brahma Reddy Battula commented on MAPREDUCE-6783: - is it dupe of MAPREDUCE-6704? > Default setting 'inheritParentEnv=false' caused mr executor failed to > initialize > > > Key: MAPREDUCE-6783 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6783 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: DENG FEI > > MR default classpath depend on the "HADOOP_MAPRED_HOME" , defined at > MRJobConfig#DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH. > On 3.0.0.alpha1 , contain "Remove parent's env vars from child processes " > which commit by [#Robert Kanter] ,but it's seem not found in jira or release > note. DefaultContainerExecutor & DefaultLinuxContinerExecutor default the > 'inheritParentEnv' as false,there is no any approach to set the > "HADOOP_MAPRED_HOME" ,the AM failed by "Could not find or load main class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516342#comment-15516342
]
Brahma Reddy Battula edited comment on MAPREDUCE-6782 at 9/27/16 4:18 AM:
--
As per internal discussion with you( that you will be working on root cause),
assigned to you.
was (Author: brahmareddy):
Assigned to you.
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: gu-chi
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6782:
Assignee: gu-chi (was: Bibin A Chundatt)
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: gu-chi
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6782) JHS task page search based on each individual column not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516342#comment-15516342
]
Brahma Reddy Battula commented on MAPREDUCE-6782:
-
Assigned to you.
> JHS task page search based on each individual column not working
>
>
> Key: MAPREDUCE-6782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: gu-chi
>
> Submit mapreduce pi job with 10 maps
> In Jobs history server selection completed job
> Select maps to Task Page for job
> Search in individual column fields
> *Expected*
> Search should be working fine in task page for individual columns
> *Actual*
> Search not working for individual column in task page
> In Attempts page the same search is working fine
> {noformat}
> jquery.dataTables.min.js:109
> Uncaught TypeError: Cannot read property 'oFeatures' of null
> fnFilter @ jquery.dataTables.min.js:109(anonymous function) @ m:49dispatch
> @ jquery-1.8.2.min.js:2h @ jquery-1.8.2.min.js:2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6682:
Target Version/s: 3.0.0-alpha2
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Fix For: 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch, MAPREDUCE-6682.03.patch, MAPREDUCE-6682.04.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6682:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Committed to trunk.. Thanks [~ajisakaa] for your contribution.
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Fix For: 3.0.0-alpha2
>
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch, MAPREDUCE-6682.03.patch, MAPREDUCE-6682.04.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405693#comment-15405693
]
Brahma Reddy Battula commented on MAPREDUCE-6682:
-
Ok.. thanks for Clarification.. will commit today..
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch, MAPREDUCE-6682.03.patch, MAPREDUCE-6682.04.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397297#comment-15397297
]
Brahma Reddy Battula commented on MAPREDUCE-6682:
-
[~ajisakaa] thanks again for updating the patch... lgtm +1 after addressing the
following..
now {{FileUtil.fullyDelete(new File(outDir.toString()));}} might not require in
{{testAbort()}}..?
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch, MAPREDUCE-6682.03.patch, MAPREDUCE-6682.04.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395692#comment-15395692
]
Brahma Reddy Battula commented on MAPREDUCE-6682:
-
[~ajisakaa] thanks for updating the patch..
one minor nit:
How about using specific outdir for this class using
{{GenericTestUtils.getTestDir(String subdir)}} and clean in teardown()..?
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch, MAPREDUCE-6682.03.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389209#comment-15389209
]
Brahma Reddy Battula commented on MAPREDUCE-6682:
-
[~ajisakaa] thanks for working on this issue and pinging me..
Approach overall looks great.
Some minor nits:
1) *{{TestMRCJCFileOutputCommitter}}:* As {{testCommitter}} also calling
{{FileUtil.fullyDelete(new File(outDir.toString()))}} at the end of
testcase,how about adding in teardown..?
2) *{{TestMRTimelineEventHandling}}:* do we need to handle {{inDir}} delete
also..?
> TestMRCJCFileOutputCommitter fails intermittently
> -
>
> Key: MAPREDUCE-6682
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Akira Ajisaka
> Attachments: MAPREDUCE-6682.00.patch, MAPREDUCE-6682.01.patch,
> MAPREDUCE-6682.02.patch
>
>
> {noformat}
> java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
> {noformat}
> *PreCommit Report*
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6402) Application Master listens on all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366104#comment-15366104 ] Brahma Reddy Battula commented on MAPREDUCE-6402: - Thanks for reply...Thought of handling the ShufflePort related in MAPREDUCE-5938 and AM IP ( as I mentioned above ) related changes here.. > Application Master listens on all IPs > - > > Key: MAPREDUCE-6402 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6402 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0 >Reporter: Peter Shi >Assignee: Varun Saxena > > The application master listening on all IPs, which may cause security > problems. Similar issue: MAPREDUCE-5938 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6402) Application Master listens on all IPs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363123#comment-15363123
]
Brahma Reddy Battula commented on MAPREDUCE-6402:
-
any update on this issue..?
I think, as per peter suggestion can we make use of nodemanger address..??
there are three places which we need consider for
{{yarn.app.mapreduce.am.job.client.port-range}}.
1) {{appContext.getNMHostname()}} can be passed in
MRClientService#serviceStart() for following two cases.
a ) {{InetSocketAddress address = new InetSocketAddress(0);}}
b) {code}
webApp =
WebApps.$for("mapreduce", AppContext.class, appContext, "ws")
.withHttpPolicy(conf, Policy.HTTP_ONLY).start(new AMWebApp());
{code}
3) "0.0.0.0" can be replace with {{context.getNMHostname()}} in
{{TaskAttempListenerImpl#startRpcServer()}}
{code}
server =
new RPC.Builder(conf).setProtocol(TaskUmbilicalProtocol.class)
.setInstance(this).setBindAddress("0.0.0.0")
.setPort(0).setNumHandlers(
conf.getInt(MRJobConfig.MR_AM_TASK_LISTENER_THREAD_COUNT,
MRJobConfig.DEFAULT_MR_AM_TASK_LISTENER_THREAD_COUNT))
.setVerbose(false).setSecretManager(jobTokenSecretManager)
.build();
{code}
Please correct me if I am wrong..
> Application Master listens on all IPs
> -
>
> Key: MAPREDUCE-6402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.0
>Reporter: Peter Shi
>Assignee: Varun Saxena
>
> The application master listening on all IPs, which may cause security
> problems. Similar issue: MAPREDUCE-5938
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6716) Shell command "mapred job -list" show improper time format
[ https://issues.apache.org/jira/browse/MAPREDUCE-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6716: Assignee: Shen Yinjie > Shell command "mapred job -list" show improper time format > --- > > Key: MAPREDUCE-6716 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6716 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.6.0, 2.8.0 > Environment: centos6.5 x64 > hadoop 2.6.0 /hadoop2.8.0 >Reporter: Shen Yinjie >Assignee: Shen Yinjie >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6716.patch, MAPREDUCE-6716_trunk.patch > > > when client excute shell "mapred job -list", > the result shows " job's startTime" is a long type，which is not > user-friendly to end-user. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5938) Shuffle port in nodemanager is binding to all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-5938: Summary: Shuffle port in nodemanager is binding to all IPs (was: AM tracking URL & Shuffle port in nodemanager is binding to all IPs ) > Shuffle port in nodemanager is binding to all IPs > -- > > Key: MAPREDUCE-5938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5938 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ashutosh Jindal >Assignee: Naganarasimha G R > Attachments: issue.jpg, mapreduce-5938.patch, result_after_fix.jpg > > > nodemanager port mapreduce.shuffle.port is listening to all ip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5938) Shuffle port in nodemanager is binding to all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361372#comment-15361372 ] Brahma Reddy Battula commented on MAPREDUCE-5938: - Thanks a lot. > Shuffle port in nodemanager is binding to all IPs > -- > > Key: MAPREDUCE-5938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5938 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ashutosh Jindal >Assignee: Naganarasimha G R > Attachments: issue.jpg, mapreduce-5938.patch, result_after_fix.jpg > > > nodemanager port mapreduce.shuffle.port is listening to all ip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-5938) AM tracking URL & Shuffle port in nodemanager is binding to all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361237#comment-15361237 ] Brahma Reddy Battula edited comment on MAPREDUCE-5938 at 7/4/16 12:36 PM: -- Any update on this issue.. As Naga suggested, can we go with new config for shuffle handler..? new config can be like *mapreduce.shuffle.address* and depreciate *mapreduce.shuffle.port* any thoughts..? So we can change defect summary..? I am not able to change the summary,[~ajisakaa] can you help on this..? can you give permission to change the defect summary. was (Author: brahmareddy): Any update on this issue.. As Naga suggested, can we go with new config for shuffle handler..? new config can be like *mapreduce.shuffle.address* and depreciate *mapreduce.shuffle.port* any thoughts..? > AM tracking URL & Shuffle port in nodemanager is binding to all IPs > > > Key: MAPREDUCE-5938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5938 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ashutosh Jindal >Assignee: Naganarasimha G R > Attachments: issue.jpg, mapreduce-5938.patch, result_after_fix.jpg > > > nodemanager port mapreduce.shuffle.port is listening to all ip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-5938) AM tracking URL & Shuffle port in nodemanager is binding to all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361237#comment-15361237 ] Brahma Reddy Battula edited comment on MAPREDUCE-5938 at 7/4/16 12:32 PM: -- Any update on this issue.. As Naga suggested, can we go with new config for shuffle handler..? new config can be like *mapreduce.shuffle.address* and depreciate *mapreduce.shuffle.port* any thoughts..? was (Author: brahmareddy): Any update on this issue.. > AM tracking URL & Shuffle port in nodemanager is binding to all IPs > > > Key: MAPREDUCE-5938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5938 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ashutosh Jindal >Assignee: Naganarasimha G R > Attachments: issue.jpg, mapreduce-5938.patch, result_after_fix.jpg > > > nodemanager port mapreduce.shuffle.port is listening to all ip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5938) AM tracking URL & Shuffle port in nodemanager is binding to all IPs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361237#comment-15361237 ] Brahma Reddy Battula commented on MAPREDUCE-5938: - Any update on this issue.. > AM tracking URL & Shuffle port in nodemanager is binding to all IPs > > > Key: MAPREDUCE-5938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5938 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ashutosh Jindal >Assignee: Naganarasimha G R > Attachments: issue.jpg, mapreduce-5938.patch, result_after_fix.jpg > > > nodemanager port mapreduce.shuffle.port is listening to all ip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320569#comment-15320569 ] Brahma Reddy Battula commented on MAPREDUCE-: - bq.Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter It's unrelated this jira and tracked MAPREDUCE-6682. check-style : indentations are inline to existing NNBench help message. if we want to fix, we may need to fix full help message. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch, > MAPREDUCE--03.patch, MAPREDUCE--04.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Attachment: MAPREDUCE--04.patch > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch, > MAPREDUCE--03.patch, MAPREDUCE--04.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Attachment: MAPREDUCE--03.patch Uploading the patch to fix the check-style issues. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch, > MAPREDUCE--03.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296245#comment-15296245 ] Brahma Reddy Battula commented on MAPREDUCE-: - Thanks for looking into this...Threads will start after barrier only.. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Attachment: MAPREDUCE--02.patch uploading the latest patch. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6676) [ NNBench ] Throw IOException when rename,delete fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246916#comment-15246916 ] Brahma Reddy Battula commented on MAPREDUCE-6676: - TestCase failures are unrelated, Raised MAPREDUCE-6681 and MAPREDUCE-6682 to track.. > [ NNBench ] Throw IOException when rename,delete fails > -- > > Key: MAPREDUCE-6676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE-6676.patch > > > Throw IOException when rename,delete fails, currently it's unknown to user > when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6682) TestMRCJCFileOutputCommitter fais intermittently
Brahma Reddy Battula created MAPREDUCE-6682:
---
Summary: TestMRCJCFileOutputCommitter fais intermittently
Key: MAPREDUCE-6682
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6682
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
{noformat}
java.lang.AssertionError: Output directory not empty expected:<0> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:153)
{noformat}
*PreCommit Report*
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6681) TestUberAM fails intermittently
Brahma Reddy Battula created MAPREDUCE-6681:
---
Summary: TestUberAM fails intermittently
Key: MAPREDUCE-6681
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6681
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
{noformat}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.verifySleepJobCounters(TestMRJobs.java:474)
at
org.apache.hadoop.mapreduce.v2.TestUberAM.verifySleepJobCounters(TestUberAM.java:71)
{noformat}
*PreCommit Build*
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6434/testReport/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6676) [ NNBench ] Throw IOException when rename,delete fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6676: Status: Patch Available (was: Open) > [ NNBench ] Throw IOException when rename,delete fails > -- > > Key: MAPREDUCE-6676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE-6676.patch > > > Throw IOException when rename,delete fails, currently it's unknown to user > when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6676) [ NNBench ] Throw IOException when rename,delete fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-6676: Attachment: MAPREDUCE-6676.patch Uploaded the patch..Kindly Review. > [ NNBench ] Throw IOException when rename,delete fails > -- > > Key: MAPREDUCE-6676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE-6676.patch > > > Throw IOException when rename,delete fails, currently it's unknown to user > when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6676) [ NNBench ] Throw IOException when rename,delete fails
Brahma Reddy Battula created MAPREDUCE-6676: --- Summary: [ NNBench ] Throw IOException when rename,delete fails Key: MAPREDUCE-6676 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6676 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Throw IOException when rename,delete fails, currently it's unknown to user when rename and delte fails.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221216#comment-15221216 ] Brahma Reddy Battula commented on MAPREDUCE-: - [~ajisakaa]/[~vinayrpet] can you please kindly review. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[
https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221023#comment-15221023
]
Brahma Reddy Battula commented on MAPREDUCE-:
-
Test failures are unrelated. MAPREDUCE-6658 is track
TestMRJobs,{{TestMRCJCFileOutputCommitter}} is passing locally. Kindly review.
> Support MultiThreads in a Map and Distribution of files in NNBench
> --
>
> Key: MAPREDUCE-
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE--01.patch
>
>
> Support Distribution of files to multiple directories generated by NNBench.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[
https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219820#comment-15219820
]
Brahma Reddy Battula commented on MAPREDUCE-:
-
Introduced the following two paramas, both are not mandatory.
{{-numberOfThreadsPerMap}} : number of threads for each map to create_write op.
default is 1.
{{-distributeFiles}} : true or false. whether files needs to be distributed in
hierarchical way to avoid all files in one directory. default is false.
> Support MultiThreads in a Map and Distribution of files in NNBench
> --
>
> Key: MAPREDUCE-
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE--01.patch
>
>
> Support Distribution of files to multiple directories generated by NNBench.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Status: Patch Available (was: Open) > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Attachment: MAPREDUCE--01.patch Uploaded the patch..Kindly Review. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6663) [NNBench] Refactor nnbench as a Tool implementation.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217478#comment-15217478 ] Brahma Reddy Battula commented on MAPREDUCE-6663: - [~ajisakaa] thanks for review and commit.and thanks [~vinayrpet] for additional review.. > [NNBench] Refactor nnbench as a Tool implementation. > > > Key: MAPREDUCE-6663 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6663 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Fix For: 2.8.0 > > Attachments: HDFS-7651-001.patch, HDFS-7651-002.patch, > HDFS-7651-003.patch, HDFS-7651-004.patch, MAPREDUCE-6663-005.patch > > > Refactor NNBench as a tool to accept generic command line option, and add a > regression test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6663) [ NN Bench ] Refactor nnbench as a Tool implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6663:
Attachment: MAPREDUCE-6663-005.patch
Uploaded the patch to address above comments.. thanks
> [ NN Bench ] Refactor nnbench as a Tool implementation.
> ---
>
> Key: MAPREDUCE-6663
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6663
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-7651-001.patch, HDFS-7651-002.patch,
> HDFS-7651-003.patch, HDFS-7651-004.patch, MAPREDUCE-6663-005.patch
>
>
> {code}
> public class NNBench {
> private static final Log LOG = LogFactory.getLog(
> "org.apache.hadoop.hdfs.NNBench");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6663) [ NN Bench ] Refactor nnbench as a Tool implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215631#comment-15215631
]
Brahma Reddy Battula commented on MAPREDUCE-6663:
-
bq.Patch generated 1 ASF License warnings
unrelated .. and it is tracked in HDFS-10221.
Test failures are unrelated.. Kindly Review.
> [ NN Bench ] Refactor nnbench as a Tool implementation.
> ---
>
> Key: MAPREDUCE-6663
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6663
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-7651-001.patch, HDFS-7651-002.patch,
> HDFS-7651-003.patch, HDFS-7651-004.patch
>
>
> {code}
> public class NNBench {
> private static final Log LOG = LogFactory.getLog(
> "org.apache.hadoop.hdfs.NNBench");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6666) Support Distribution of files to multiple directories generated by NNBench
Brahma Reddy Battula created MAPREDUCE-: --- Summary: Support Distribution of files to multiple directories generated by NNBench Key: MAPREDUCE- URL: https://issues.apache.org/jira/browse/MAPREDUCE- Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6663) [ NN Bench ] Refactor nnbench as a Tool implementation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula moved HDFS-7651 to MAPREDUCE-6663:
---
Affects Version/s: (was: 2.6.0)
2.6.0
Key: MAPREDUCE-6663 (was: HDFS-7651)
Project: Hadoop Map/Reduce (was: Hadoop HDFS)
> [ NN Bench ] Refactor nnbench as a Tool implementation.
> ---
>
> Key: MAPREDUCE-6663
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6663
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-7651-001.patch, HDFS-7651-002.patch,
> HDFS-7651-003.patch, HDFS-7651-004.patch
>
>
> {code}
> public class NNBench {
> private static final Log LOG = LogFactory.getLog(
> "org.apache.hadoop.hdfs.NNBench");
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155298#comment-15155298
]
Brahma Reddy Battula commented on MAPREDUCE-6637:
-
[~andrew.wang] thanks for review and commit.
> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
>
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 2.7.3
>
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6637:
Component/s: test
> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
>
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6637:
Status: Patch Available (was: Open)
> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
>
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
[
https://issues.apache.org/jira/browse/MAPREDUCE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated MAPREDUCE-6637:
Attachment: MAPREDUCE-6637.patch
> Testcase Failure : TestFileInputFormat.testSplitLocationInfo
>
>
> Key: MAPREDUCE-6637
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: MAPREDUCE-6637.patch
>
>
> Following testcase is failing after HADOOP-12810
> {noformat}
> FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
> Error Message:
> expected:<2> but was:<1>
> Stack Trace:
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6637) Testcase Failure : TestFileInputFormat.testSplitLocationInfo
Brahma Reddy Battula created MAPREDUCE-6637:
---
Summary: Testcase Failure :
TestFileInputFormat.testSplitLocationInfo
Key: MAPREDUCE-6637
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6637
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Following testcase is failing after HADOOP-12810
{noformat}
FAILED: org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo[0]
Error Message:
expected:<2> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<2> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestFileInputFormat.testSplitLocationInfo(TestFileInputFormat.java:115)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136785#comment-15136785
]
Brahma Reddy Battula commented on MAPREDUCE-6626:
-
[~linyiqun] latest patch looks good to me. [~ajisakaa] what do you think.?
> Reuse objectMapper instance in MapReduce
>
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch,
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of
> instances) for data binding; many optimizations for reuse (of symbol tables,
> some buffers) depend on ObjectMapper instances being reused.
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+=4=ja=clnk=jp,
> it's similar to HDFS-9724.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
