[jira] [Work logged] (MAPREDUCE-7370) Parallelize MultipleOutputs#close call
[
https://issues.apache.org/jira/browse/MAPREDUCE-7370?focusedWorklogId=783691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783691
]
ASF GitHub Bot logged work on MAPREDUCE-7370:
-
Author: ASF GitHub Bot
Created on: 22/Jun/22 04:19
Start Date: 22/Jun/22 04:19
Worklog Time Spent: 10m
Work Description: PrabhuJoseph commented on PR #4248:
URL: https://github.com/apache/hadoop/pull/4248#issuecomment-1162620418
The Patch looks good. @steveloughran Do you want to take a quick look on the
latest patch to make sure it meets the requirement of this Jira you reported.
Issue Time Tracking
---
Worklog Id: (was: 783691)
Time Spent: 1h (was: 50m)
> Parallelize MultipleOutputs#close call
> --
>
> Key: MAPREDUCE-7370
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7370
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This call takes more time when there are lot of files to close and there is a
> high latency to close. Parallelize MultipleOutputs#close call to improve the
> speed.
> {code}
> public void close() throws IOException {
> for (RecordWriter writer : recordWriters.values()) {
> writer.close(null);
> }
> }
> {code}
> Idea is from [~ste...@apache.org]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7026) Shuffle Fetcher does not log the actual error message thrown by ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned MAPREDUCE-7026:
-
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> Shuffle Fetcher does not log the actual error message thrown by ShuffleHandler
> --
>
> Key: MAPREDUCE-7026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: supportability
> Attachments: MAPREDUCE-7026.1.patch, MAPREDUCE-7026.2.patch,
> MAPREDUCE-7026.3.patch
>
>
> A job is failing with reduce tasks failed to fetch map output and the
> NodeManager ShuffleHandler failed to serve the map outputs with some
> IOException like below. ShuffleHandler sends the actual error message in
> response inside sendError() but the Fetcher does not log this message.
> Logs from NodeManager ShuffleHandler:
> {code}
> 2017-12-18 10:10:30,728 ERROR mapred.ShuffleHandler
> (ShuffleHandler.java:messageReceived(962)) - Shuffle error in populating
> headers :
> java.io.IOException: Error Reading IndexFile
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1089)
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:958)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Owner 'hbase' for path
> /grid/7/hadoop/yarn/local/usercache/bde/appcache/application_1512457770852_9447/output/attempt_1512457770852_9447_1_01_07_0_10004/file.out.index
> did not match expected owner 'bde'
> at
[jira] [Commented] (MAPREDUCE-7026) Shuffle Fetcher does not log the actual error message thrown by ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-7026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557210#comment-17557210
]
Ashutosh Gupta commented on MAPREDUCE-7026:
---
The issue seems to be pending for a while. Taking it up.
> Shuffle Fetcher does not log the actual error message thrown by ShuffleHandler
> --
>
> Key: MAPREDUCE-7026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Labels: supportability
> Attachments: MAPREDUCE-7026.1.patch, MAPREDUCE-7026.2.patch,
> MAPREDUCE-7026.3.patch
>
>
> A job is failing with reduce tasks failed to fetch map output and the
> NodeManager ShuffleHandler failed to serve the map outputs with some
> IOException like below. ShuffleHandler sends the actual error message in
> response inside sendError() but the Fetcher does not log this message.
> Logs from NodeManager ShuffleHandler:
> {code}
> 2017-12-18 10:10:30,728 ERROR mapred.ShuffleHandler
> (ShuffleHandler.java:messageReceived(962)) - Shuffle error in populating
> headers :
> java.io.IOException: Error Reading IndexFile
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.populateHeaders(ShuffleHandler.java:1089)
> at
> org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:958)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
> at
> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Owner 'hbase' for path
> /grid/7/hadoop/yarn/local/usercache/bde/appcache/application_1512457770852_9447/output/attempt_1512457770852_9447_1_01_07_0_10004/file.out.index
> did not
[jira] [Commented] (MAPREDUCE-7201) Make Job History File Permissions configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557204#comment-17557204 ] Ashutosh Gupta commented on MAPREDUCE-7201: --- The issue seems to be pending for a while. Taking it up to make required changes and raising PR > Make Job History File Permissions configurable > -- > > Key: MAPREDUCE-7201 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7201 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: MAPREDUCE-7201-001.patch > > > Job History File Permissions are hardcoded to 770. MAPREDUCE-7010 allows to > configure the intermediate user directory permission but still the jhist file > permission are not changed. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7201) Make Job History File Permissions configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned MAPREDUCE-7201: - Assignee: Ashutosh Gupta (was: Prabhu Joseph) > Make Job History File Permissions configurable > -- > > Key: MAPREDUCE-7201 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7201 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Ashutosh Gupta >Priority: Major > Attachments: MAPREDUCE-7201-001.patch > > > Job History File Permissions are hardcoded to 770. MAPREDUCE-7010 allows to > configure the intermediate user directory permission but still the jhist file > permission are not changed. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[
https://issues.apache.org/jira/browse/MAPREDUCE-7391?focusedWorklogId=783591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783591
]
ASF GitHub Bot logged work on MAPREDUCE-7391:
-
Author: ASF GitHub Bot
Created on: 21/Jun/22 22:09
Start Date: 21/Jun/22 22:09
Worklog Time Spent: 10m
Work Description: mukund-thakur commented on code in PR #4472:
URL: https://github.com/apache/hadoop/pull/4472#discussion_r903105616
##
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java:
##
@@ -44,22 +45,31 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FilterFileSystem;
+import org.apache.hadoop.fs.FutureDataInputStreamBuilder;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PositionedReadable;
import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.fs.impl.FutureDataInputStreamBuilderImpl;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.MRConfig;
import org.apache.hadoop.mapreduce.MRJobConfig;
+import org.apache.hadoop.util.functional.CallableRaisingIOE;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.stubbing.Answer;
+/**
+ * Test the LocalDistributedCacheManager using mocking.
+ * This suite is brittle to changes in the class under test.
+ */
@SuppressWarnings("deprecation")
public class TestLocalDistributedCacheManager {
+ public static final byte[] TEST_DATA = "This is a test file\n".getBytes();
Review Comment:
nit: private?
Issue Time Tracking
---
Worklog Id: (was: 783591)
Time Spent: 50m (was: 40m)
> TestLocalDistributedCacheManager failing after HADOOP-16202
> ---
>
> Key: MAPREDUCE-7391
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7391
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0, 3.3.9
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing
> with an NPE
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close
[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557104#comment-17557104
]
Ashutosh Gupta commented on MAPREDUCE-7369:
---
[~cnauroth] - I have another Jira as well related to this. Can you review it?
Thanks.
JIRA: https://issues.apache.org/jira/browse/MAPREDUCE-7370
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
> Attachments: MAPREDUCE-7369.001.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close
[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557094#comment-17557094
]
Chris Nauroth commented on MAPREDUCE-7369:
--
[~groot] , thank you for the contribution. [~iwasakims] , thank you for the
final review and commit.
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
> Attachments: MAPREDUCE-7369.001.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[ https://issues.apache.org/jira/browse/MAPREDUCE-7391?focusedWorklogId=783489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783489 ] ASF GitHub Bot logged work on MAPREDUCE-7391: - Author: ASF GitHub Bot Created on: 21/Jun/22 17:29 Start Date: 21/Jun/22 17:29 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4472: URL: https://github.com/apache/hadoop/pull/4472#issuecomment-1162060152 @ayushtkn @mukund-thakur could you two review this. test failure on trunk/branch-3.3 Issue Time Tracking --- Worklog Id: (was: 783489) Time Spent: 40m (was: 0.5h) > TestLocalDistributedCacheManager failing after HADOOP-16202 > --- > > Key: MAPREDUCE-7391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7391 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing > with an NPE -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7389) Typo in description of "mapreduce.application.classpath" in mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556919#comment-17556919
]
Ayush Saxena commented on MAPREDUCE-7389:
-
Committed to trunk.
Thanx [~c.bartolomaeus] for the contribution!!!
> Typo in description of "mapreduce.application.classpath" in mapred-default.xml
> --
>
> Key: MAPREDUCE-7389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.3
>Reporter: Christian Bartolomäus
>Assignee: Christian Bartolomäus
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> There is a small typo for {variable} in the description of
> "mapreduce.application.classpath" in
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml.
> {noformat}
> If mapreduce.app-submission.cross-platform is false, platform-specific
> environment vairable expansion syntax would be used to construct the default
> CLASSPATH entries.
> {noformat}
> I just stumbled upon that and will open a PR over at github.com.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7389) Typo in description of "mapreduce.application.classpath" in mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved MAPREDUCE-7389.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Typo in description of "mapreduce.application.classpath" in mapred-default.xml
> --
>
> Key: MAPREDUCE-7389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.3
>Reporter: Christian Bartolomäus
>Assignee: Christian Bartolomäus
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> There is a small typo for {variable} in the description of
> "mapreduce.application.classpath" in
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml.
> {noformat}
> If mapreduce.app-submission.cross-platform is false, platform-specific
> environment vairable expansion syntax would be used to construct the default
> CLASSPATH entries.
> {noformat}
> I just stumbled upon that and will open a PR over at github.com.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7389) Typo in description of "mapreduce.application.classpath" in mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-7389?focusedWorklogId=783393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783393
]
ASF GitHub Bot logged work on MAPREDUCE-7389:
-
Author: ASF GitHub Bot
Created on: 21/Jun/22 13:55
Start Date: 21/Jun/22 13:55
Worklog Time Spent: 10m
Work Description: ayushtkn commented on PR #4440:
URL: https://github.com/apache/hadoop/pull/4440#issuecomment-1161782438
Not me, but @goiri is. I am myself a beneficiary. :-)
Thanx everyone. Merged!!!
Issue Time Tracking
---
Worklog Id: (was: 783393)
Time Spent: 1h 50m (was: 1h 40m)
> Typo in description of "mapreduce.application.classpath" in mapred-default.xml
> --
>
> Key: MAPREDUCE-7389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.3
>Reporter: Christian Bartolomäus
>Assignee: Christian Bartolomäus
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> There is a small typo for {variable} in the description of
> "mapreduce.application.classpath" in
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml.
> {noformat}
> If mapreduce.app-submission.cross-platform is false, platform-specific
> environment vairable expansion syntax would be used to construct the default
> CLASSPATH entries.
> {noformat}
> I just stumbled upon that and will open a PR over at github.com.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7389) Typo in description of "mapreduce.application.classpath" in mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-7389?focusedWorklogId=783392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783392
]
ASF GitHub Bot logged work on MAPREDUCE-7389:
-
Author: ASF GitHub Bot
Created on: 21/Jun/22 13:54
Start Date: 21/Jun/22 13:54
Worklog Time Spent: 10m
Work Description: ayushtkn merged PR #4440:
URL: https://github.com/apache/hadoop/pull/4440
Issue Time Tracking
---
Worklog Id: (was: 783392)
Time Spent: 1h 40m (was: 1.5h)
> Typo in description of "mapreduce.application.classpath" in mapred-default.xml
> --
>
> Key: MAPREDUCE-7389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.3
>Reporter: Christian Bartolomäus
>Assignee: Christian Bartolomäus
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There is a small typo for {variable} in the description of
> "mapreduce.application.classpath" in
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml.
> {noformat}
> If mapreduce.app-submission.cross-platform is false, platform-specific
> environment vairable expansion syntax would be used to construct the default
> CLASSPATH entries.
> {noformat}
> I just stumbled upon that and will open a PR over at github.com.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7389) Typo in description of "mapreduce.application.classpath" in mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reassigned MAPREDUCE-7389:
---
Assignee: Christian Bartolomäus
> Typo in description of "mapreduce.application.classpath" in mapred-default.xml
> --
>
> Key: MAPREDUCE-7389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.3
>Reporter: Christian Bartolomäus
>Assignee: Christian Bartolomäus
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There is a small typo for {variable} in the description of
> "mapreduce.application.classpath" in
> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml.
> {noformat}
> If mapreduce.app-submission.cross-platform is false, platform-specific
> environment vairable expansion syntax would be used to construct the default
> CLASSPATH entries.
> {noformat}
> I just stumbled upon that and will open a PR over at github.com.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7392) bug for GzipCodec in native task
chenfengge created MAPREDUCE-7392: - Summary: bug for GzipCodec in native task Key: MAPREDUCE-7392 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7392 Project: Hadoop Map/Reduce Issue Type: Bug Components: nativetask Reporter: chenfengge perpare data command: yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar teragen -D mapreduce.job.maps=722 -D mapreduce.job.reduces=500 110 /testinput test command: yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar terasort -Dmapreduce.map.output.compress=true -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec -Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator -D mapreduce.job.reduces=500 /testinput /testoutput test will blocked. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[
https://issues.apache.org/jira/browse/MAPREDUCE-7391?focusedWorklogId=783329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-783329
]
ASF GitHub Bot logged work on MAPREDUCE-7391:
-
Author: ASF GitHub Bot
Created on: 21/Jun/22 11:09
Start Date: 21/Jun/22 11:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4472:
URL: https://github.com/apache/hadoop/pull/4472#issuecomment-1161599824
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common:
The patch generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-mapreduce-client-common
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 93m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4472/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4472 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a56e443599a8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 698c70e9e71e3c34c9713d56dcc7392f44dcbcc4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4472/2/testReport/ |
| Max. process+thread count | 621 (vs. ulimit of 5500) |
| modules | C:
