from:"Szilard Nemeth \(Jira\)"

[jira] [Resolved] (MAPREDUCE-7456) Extend add-opens flag to container launch commands on JDK17 nodes

2023-09-27 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth resolved MAPREDUCE-7456.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Extend add-opens flag to container launch commands on JDK17 nodes
> -
>
> Key: MAPREDUCE-7456
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7456
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Affects Versions: 3.4.0
>Reporter: Peter Szucs
>Assignee: Peter Szucs
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> There was a previous ticket for adding add-opens flag to container launch to 
> be able to run them on JDK17 nodes: 
> https://issues.apache.org/jira/browse/MAPREDUCE-7449
> As testing discovered, this should be extended with 
> "{_}-add-exports=java.base/sun.net.dns=ALL-UNNAMED{_}" and 
> "{_}-add-exports=java.base/sun.net.util=ALL-UNNAMED{_}" options to be able to 
> run containers on Isilon.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7441) Race condition in closing FadvisedFileRegion

2023-06-23 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth resolved MAPREDUCE-7441.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Race condition in closing FadvisedFileRegion
> 
>
> Key: MAPREDUCE-7441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7441
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> This issue is similar to the one described in MAPREDUCE-7095, just for 
> FadvisedFileRegion.transferSuccessful. There are warning messages when 
> multiple threads are calling the transferSuccessful method:
> {code:java}
> 2023-05-25 08:41:57,288 WARN org.apache.hadoop.mapred.FadvisedFileRegion: 
> Failed to manage OS cache for 
> /hadoop/data04/yarn/nm/usercache/hive/appcache/application_1684916804740_8245/output/attempt_1684916804740_8245_1_00_001154_0_10003/file.out
> EBADF: Bad file descriptor
> at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
> at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:271)
> at 
> org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:148)
> at 
> org.apache.hadoop.mapred.FadvisedFileRegion.transferSuccessful(FadvisedFileRegion.java:163)
> at 
> org.apache.hadoop.mapred.ShuffleChannelHandler.lambda$sendMapOutput$0(ShuffleChannelHandler.java:516)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
> at 
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7095) Race conditions in closing FadvisedChunkedFile

2023-06-23 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7095:
--
Description: 
When a file is closed multiple times by multiple threads, all but the first 
close will generate a WARNING message.
{code:java}
11:04:33.605 AM WARNFadvisedChunkedFile 
Failed to manage OS cache for 
/var/run/100/yarn/nm/usercache/systest/appcache/application_1521665017379_0062/output/attempt_1521665017379_0062_m_012797_0/file.out
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native 
Method)
at 
org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
at 
org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
at 
org.apache.hadoop.mapred.FadvisedChunkedFile.close(FadvisedChunkedFile.java:76)
at 
org.jboss.netty.handler.stream.ChunkedWriteHandler.closeInput(ChunkedWriteHandler.java:303)
at 
org.jboss.netty.handler.stream.ChunkedWriteHandler.discard(ChunkedWriteHandler.java:163)
at 
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:192)
at 
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:137)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelClosed(SimpleChannelUpstreamHandler.java:225)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570)
at 
org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493)
at 
org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371)
at 
org.jboss.netty.handler.ssl.SslHandler.channelClosed(SslHandler.java:1667)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:375)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at 
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at 
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at 
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}

  was:
When a file is closed mutple times by multiple threads, all but the first close 
will generate a WARNING message.
{code:java}
11:04:33.605 AM WARNFadvisedChunkedFile 
Failed to manage OS cache for 
/var/run/100/yarn/nm/usercache/systest/appcache/application_1521665017379_0062/output/attempt_1521665017379_0062_m_012797_0/file.out
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native 
Method)
at 
org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
at

[jira] [Updated] (MAPREDUCE-7434) Fix ShuffleHandler tests

2023-03-01 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7434:
--
Fix Version/s: 3.4.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Fix ShuffleHandler tests
> 
>
> Key: MAPREDUCE-7434
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7434
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1143/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testMapFileAccess/
> {code}
> Error Message
> Server returned HTTP response code: 500 for URL: 
> http://127.0.0.1:13562/mapOutput?job=job_1_0001=0=attempt_1_0001_m_01_0
> Stacktrace
> java.io.IOException: Server returned HTTP response code: 500 for URL: 
> http://127.0.0.1:13562/mapOutput?job=job_1_0001=0=attempt_1_0001_m_01_0
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1902)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
>   at 
> org.apache.hadoop.mapred.TestShuffleHandler.testMapFileAccess(TestShuffleHandler.java:292)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at java.lang.Thread.run(Thread.java:750)
> Standard Output
> 12:04:17.466 [Time-limited test] DEBUG o.a.h.m.lib.MutableMetricsFactory - 
> field org.apache.hadoop.metrics2.lib.MutableGaugeInt 
> org.apache.hadoop.mapred.ShuffleHandler$ShuffleMetrics.shuffleConnections 
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, 
> sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[# 
> of current shuffle connections])
> 12:04:17.466 [Time-limited test] DEBUG o.a.h.m.lib.MutableMetricsFactory - 
> field org.apache.hadoop.metrics2.lib.MutableCounterLong 
> org.apache.hadoop.mapred.ShuffleHandler$ShuffleMetrics.shuffleOutputBytes 
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, 
> sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, 
> value=[Shuffle output in bytes])
> 12:04:17.466 [Time-limited test] DEBUG o.a.h.m.lib.MutableMetricsFactory - 
> field org.apache.hadoop.metrics2.lib.MutableCounterInt 
> org.apache.hadoop.mapred.ShuffleHandler$ShuffleMetrics.shuffleOutputsFailed 
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, 
> sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[# 
> of failed shuffle outputs])
> 12:04:17.466 [Time-limited test] DEBUG o.a.h.m.lib.MutableMetricsFactory - 
> field org.apache.hadoop.metrics2.lib.MutableCounterInt 
> org.apache.hadoop.mapred.ShuffleHandler$ShuffleMetrics.shuffleOutputsOK with 
> annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, 
> sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[# 
> of succeeeded shuffle outputs])
> 12:04:17.466 [Time-limited test] DEBUG o.a.h.m.impl.MetricsSystemImpl - 
> ShuffleMetrics, Shuffle output metrics
> 12:04:17.467 [Time-limited test] DEBUG o.a.hadoop.service.AbstractService - 
> Service: mapreduce_shuffle entered state INITED
> 12:04:17.477 [Time-limited test] DEBUG o.a.hadoop.service.AbstractService - 
> Config has been overridden during init
> 12:04:17.478 [Time-limited test] INFO  org.apache.hadoop.mapred.IndexCache - 
> IndexCache created with max memory = 10485760
> 12:04:17.479 [Time-limited test] DEBUG o.a.h.m.lib.MutableMetricsFactory - 
> field org.apache.hadoop.metrics2.lib.MutableGaugeInt 
> org.apache.hadoop.mapred.ShuffleHandler$ShuffleMetrics.shuffleConnections 
> with

[jira] [Commented] (MAPREDUCE-7431) ShuffleHandler is not working correctly in SSL mode after the Netty 4 upgrade

2023-01-11 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17675544#comment-17675544
 ] 

Szilard Nemeth commented on MAPREDUCE-7431:
---

Thanks [~tdomok] for the thorough description, very helpful and detailed.
Thumbs up!

> ShuffleHandler is not working correctly in SSL mode after the Netty 4 upgrade
> -
>
> Key: MAPREDUCE-7431
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7431
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Attachments: sendMapPipeline.png
>
>
> HADOOP-15327 introduced some regressions in the ShuffleHandler.
> h3. 1. a memory leak
> {code:java}
> ERROR io.netty.util.ResourceLeakDetector: LEAK: ByteBuf.release() was not 
> called before it's garbage-collected. See 
> https://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
>  
> The Shuffle's channelRead didn't release the message properly, the fix would 
> be this:
> {code:java}
>   try {
> // 
>   } finally {
> ReferenceCountUtil.release(msg);
>   }
> {code}
> Or even simpler:
> {code:java}
> extends SimpleChannelInboundHandler
> {code}
> h3. 1. a bug in SSL mode with more than 1 reducers
> It manifested in multiple errors:
> {code:java}
> ERROR org.apache.hadoop.mapred.ShuffleHandler: Future is unsuccessful. Cause:
> java.io.IOException: Broken pipe
> ERROR org.apache.hadoop.mapred.ShuffleHandler: Future is unsuccessful. Cause:
> java.nio.channels.ClosedChannelException
> // if the reducer memory was not enough, then even this:
> Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in 
> shuffle in fetcher#2
>   at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>   at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
>   at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
>   at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
>   at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> {code}
> *Configuration* - mapred-site.xml
> {code:java}
> mapreduce.shuffle.ssl.enabled=true
> {code}
> Alternative is to build a custom jar where *FadvisedFileRegion* is replaced 
> with *FadvisedChunkedFile* in {*}sendMapOutput{*}.
> *Reproduction*
> {code:java}
> hdfs dfs -rm -r -skipTrash /tmp/sort_input
> hdfs dfs -rm -r -skipTrash /tmp/sort_output
> yarn jar 
> hadoop-3.4.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar
>  randomwriter "-Dmapreduce.randomwriter.totalbytes=100" 
> /tmp/sort_input
> yarn jar 
> hadoop-3.4.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar
>  sort -Dmapreduce.job.reduce.slowstart.completedmaps=1 -r 40 /tmp/sort_input 
> /tmp/sort_output | tee sort_app_output.txt
> {code}
> h3. ShuffleHandler's protocol
> {code:java}
> // HTTP Request
> GET 
> /mapOutput?job=job_1672901779104_0001=0=attempt_1672901779104_0001_m_03_0,attempt_1672901779104_0001_m_02_0,attempt_1672901779104_0001_m_01_0,attempt_1672901779104_0001_m_00_0,attempt_1672901779104_0001_m_05_0,attempt_1672901779104_0001_m_12_0,attempt_1672901779104_0001_m_09_0,attempt_1672901779104_0001_m_10_0,attempt_1672901779104_0001_m_07_0,attempt_1672901779104_0001_m_11_0,attempt_1672901779104_0001_m_08_0,attempt_1672901779104_0001_m_13_0,attempt_1672901779104_0001_m_14_0,attempt_1672901779104_0001_m_15_0,attempt_1672901779104_0001_m_19_0,attempt_1672901779104_0001_m_18_0,attempt_1672901779104_0001_m_16_0,attempt_1672901779104_0001_m_17_0,attempt_1672901779104_0001_m_20_0,attempt_1672901779104_0001_m_23_0
>  HTTP/1.1
> + keep alive headers
> // HTTP Response Headers
> content-length=sum(serialised ShuffleHeader in bytes + MapOutput size)
> + keep alive headers
> // Response Data (transfer-encoding=chunked)
> serialised ShuffleHeader
> content of the MapOutput file

[jira] [Updated] (MAPREDUCE-7431) ShuffleHandler is not working correctly in SSL mode after the Netty 4 upgrade

2023-01-10 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7431:
--
Status: Patch Available  (was: Open)

> ShuffleHandler is not working correctly in SSL mode after the Netty 4 upgrade
> -
>
> Key: MAPREDUCE-7431
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7431
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Priority: Major
> Attachments: sendMapPipeline.png
>
>
> HADOOP-15327 introduced some regressions in the ShuffleHandler.
> h3. 1. a memory leak
> {code:java}
> ERROR io.netty.util.ResourceLeakDetector: LEAK: ByteBuf.release() was not 
> called before it's garbage-collected. See 
> https://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
>  
> The Shuffle's channelRead didn't release the message properly, the fix would 
> be this:
> {code:java}
>   try {
> // 
>   } finally {
> ReferenceCountUtil.release(msg);
>   }
> {code}
> Or even simpler:
> {code:java}
> extends SimpleChannelInboundHandler
> {code}
> h3. 1. a bug in SSL mode with more than 1 reducers
> It manifested in multiple errors:
> {code:java}
> ERROR org.apache.hadoop.mapred.ShuffleHandler: Future is unsuccessful. Cause:
> java.io.IOException: Broken pipe
> ERROR org.apache.hadoop.mapred.ShuffleHandler: Future is unsuccessful. Cause:
> java.nio.channels.ClosedChannelException
> // if the reducer memory was not enough, then even this:
> Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in 
> shuffle in fetcher#2
>   at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>   at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
>   at 
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
>   at 
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
>   at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210)
>   at 
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91)
> {code}
> *Configuration* - mapred-site.xml
> {code:java}
> mapreduce.shuffle.ssl.enabled=true
> {code}
> Alternative is to build a custom jar where *FadvisedFileRegion* is replaced 
> with *FadvisedChunkedFile* in {*}sendMapOutput{*}.
> *Reproduction*
> {code:java}
> hdfs dfs -rm -r -skipTrash /tmp/sort_input
> hdfs dfs -rm -r -skipTrash /tmp/sort_output
> yarn jar 
> hadoop-3.4.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar
>  randomwriter "-Dmapreduce.randomwriter.totalbytes=100" 
> /tmp/sort_input
> yarn jar 
> hadoop-3.4.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar
>  sort -Dmapreduce.job.reduce.slowstart.completedmaps=1 -r 40 /tmp/sort_input 
> /tmp/sort_output | tee sort_app_output.txt
> {code}
> h3. ShuffleHandler's protocol
> {code:java}
> // HTTP Request
> GET 
> /mapOutput?job=job_1672901779104_0001=0=attempt_1672901779104_0001_m_03_0,attempt_1672901779104_0001_m_02_0,attempt_1672901779104_0001_m_01_0,attempt_1672901779104_0001_m_00_0,attempt_1672901779104_0001_m_05_0,attempt_1672901779104_0001_m_12_0,attempt_1672901779104_0001_m_09_0,attempt_1672901779104_0001_m_10_0,attempt_1672901779104_0001_m_07_0,attempt_1672901779104_0001_m_11_0,attempt_1672901779104_0001_m_08_0,attempt_1672901779104_0001_m_13_0,attempt_1672901779104_0001_m_14_0,attempt_1672901779104_0001_m_15_0,attempt_1672901779104_0001_m_19_0,attempt_1672901779104_0001_m_18_0,attempt_1672901779104_0001_m_16_0,attempt_1672901779104_0001_m_17_0,attempt_1672901779104_0001_m_20_0,attempt_1672901779104_0001_m_23_0
>  HTTP/1.1
> + keep alive headers
> // HTTP Response Headers
> content-length=sum(serialised ShuffleHeader in bytes + MapOutput size)
> + keep alive headers
> // Response Data (transfer-encoding=chunked)
> serialised ShuffleHeader
> content of the MapOutput file (start offset - length)
> serialised ShuffleHeader
> content of the MapOutput file (start offset - length)
> serialised

[jira] [Updated] (MAPREDUCE-7379) RMContainerRequestor#makeRemoteRequest has confusing log message

2022-05-11 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7379:
--
Hadoop Flags: Reviewed
  Resolution: Fixed
  Status: Resolved  (was: Patch Available)

> RMContainerRequestor#makeRemoteRequest has confusing log message
> 
>
> Key: MAPREDUCE-7379
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7379
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Ashutosh Gupta
>Priority: Trivial
>  Labels: newbie, newbie++, pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-9355.001.patch, YARN-9355.002.patch
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#makeRemoteRequest 
> has this log: 
> {code:java}
> if (ask.size() > 0 || release.size() > 0) {
>   LOG.info("getResources() for " + applicationId + ":" + " ask="
>   + ask.size() + " release= " + release.size() + " newContainers="
>   + allocateResponse.getAllocatedContainers().size()
>   + " finishedContainers=" + numCompletedContainers
>   + " resourcelimit=" + availableResources + " knownNMs="
>   + clusterNmCount);
> }
> {code}
> The reason why "getResources()" is printed because 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator#getResources 
> invokes makeRemoteRequest. This is not too informative and error-prone as 
> name of getResources could change over time and the log will be outdated. 
> Moreover, it's not a good idea to print a method name from a method below the 
> current one in the stack.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7379) RMContainerRequestor#makeRemoteRequest has confusing log message

2022-05-11 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7379:
--
Fix Version/s: 3.4.0

> RMContainerRequestor#makeRemoteRequest has confusing log message
> 
>
> Key: MAPREDUCE-7379
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7379
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Ashutosh Gupta
>Priority: Trivial
>  Labels: newbie, newbie++, pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-9355.001.patch, YARN-9355.002.patch
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#makeRemoteRequest 
> has this log: 
> {code:java}
> if (ask.size() > 0 || release.size() > 0) {
>   LOG.info("getResources() for " + applicationId + ":" + " ask="
>   + ask.size() + " release= " + release.size() + " newContainers="
>   + allocateResponse.getAllocatedContainers().size()
>   + " finishedContainers=" + numCompletedContainers
>   + " resourcelimit=" + availableResources + " knownNMs="
>   + clusterNmCount);
> }
> {code}
> The reason why "getResources()" is printed because 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator#getResources 
> invokes makeRemoteRequest. This is not too informative and error-prone as 
> name of getResources could change over time and the log will be outdated. 
> Moreover, it's not a good idea to print a method name from a method below the 
> current one in the stack.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Moved] (MAPREDUCE-7379) RMContainerRequestor#makeRemoteRequest has confusing log message

2022-05-11 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth moved YARN-9355 to MAPREDUCE-7379:
-

Key: MAPREDUCE-7379  (was: YARN-9355)
Project: Hadoop Map/Reduce  (was: Hadoop YARN)

> RMContainerRequestor#makeRemoteRequest has confusing log message
> 
>
> Key: MAPREDUCE-7379
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7379
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Ashutosh Gupta
>Priority: Trivial
>  Labels: newbie, newbie++, pull-request-available
> Attachments: YARN-9355.001.patch, YARN-9355.002.patch
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#makeRemoteRequest 
> has this log: 
> {code:java}
> if (ask.size() > 0 || release.size() > 0) {
>   LOG.info("getResources() for " + applicationId + ":" + " ask="
>   + ask.size() + " release= " + release.size() + " newContainers="
>   + allocateResponse.getAllocatedContainers().size()
>   + " finishedContainers=" + numCompletedContainers
>   + " resourcelimit=" + availableResources + " knownNMs="
>   + clusterNmCount);
> }
> {code}
> The reason why "getResources()" is printed because 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator#getResources 
> invokes makeRemoteRequest. This is not too informative and error-prone as 
> name of getResources could change over time and the log will be outdated. 
> Moreover, it's not a good idea to print a method name from a method below the 
> current one in the stack.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2022-03-21 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7266:
--
Description: 
"historyContext" class attribute at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
 is assigned a cast of another class attribute - "jobHistoryService" - 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
 however it does not need to be stored separately because it is only ever used 
once in the class, and that too as an argument while instantiating the 
HistoryClientService class at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.

Therefore, we could just delete the lines at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
 and 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
 completely and instantiate the HistoryClientService as follows:

{code}
  @VisibleForTesting
  protected HistoryClientService createHistoryClientService() {
return new HistoryClientService((HistoryContext)jobHistoryService, 
this.jhsDTSecretManager);
  }
{code}

  was:
"historyContext" class attribute at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
 is assigned a cast of another class attribute - "jobHistoryService" - 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
 however it does not need to be stored separately because it is only ever used 
once in the clas, and that too as an argument while instantiating the 
HistoryClientService class at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.

Therefore, we could just delete the lines at 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
 and 
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
 completely and instantiate the HistoryClientService as follows:

{code}
  @VisibleForTesting
  protected HistoryClientService createHistoryClientService() {
return new HistoryClientService((HistoryContext)jobHistoryService, 
this.jhsDTSecretManager);
  }
{code}


> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.3.0, 3.2.2
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the class, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
>

[jira] [Updated] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config

2020-11-24 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7309:
--
Fix Version/s: 3.4.0

> Improve performance of reading resource request for mapper/reducers from 
> config
> ---
>
> Key: MAPREDUCE-7309
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
>Reporter: Wangda Tan
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309-004.patch, 
> MAPREDUCE-7309-005.patch, MAPREDUCE-7309.001.patch, MAPREDUCE-7309.002.patch
>
>
> This is an issue could affect all the releases which includes YARN-6927. 
> Basically, we use regex match repeatedly when we read mapper/reducer resource 
> request from config files. When we have large config file, and large number 
> of splits, it could take a long time.  
> We saw AM could take hours to parse config when we have 200k+ splits, with a 
> large config file (hundreds of kbs). 
> We should do proper caching for pre-configured resource requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config

2020-11-24 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238235#comment-17238235
 ] 

Szilard Nemeth commented on MAPREDUCE-7309:
---

Thanks [~pbacsko] & [~wangda] for working on this.

Latest patch LGTM, committed to trunk.

 

[~pbacsko] Do you want to backport this to 3.2 and 3.1? Especially asking this 
because the description mentions all releases are affected that has YARN-6927.

Also, can you add some info of the problematic method 
(org.apache.hadoop.conf.Configuration#getValByRegex) and the way it is used by 
TaskAttemptImpl to the description of this jira?

Thanks

> Improve performance of reading resource request for mapper/reducers from 
> config
> ---
>
> Key: MAPREDUCE-7309
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
>Reporter: Wangda Tan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309-004.patch, 
> MAPREDUCE-7309-005.patch, MAPREDUCE-7309.001.patch, MAPREDUCE-7309.002.patch
>
>
> This is an issue could affect all the releases which includes YARN-6927. 
> Basically, we use regex match repeatedly when we read mapper/reducer resource 
> request from config files. When we have large config file, and large number 
> of splits, it could take a long time.  
> We saw AM could take hours to parse config when we have 200k+ splits, with a 
> large config file (hundreds of kbs). 
> We should do proper caching for pre-configured resource requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config

2020-11-24 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7309:
--
Description: 
This is an issue could affect all the releases which includes YARN-6927. 

Basically, we use regex match repeatedly when we read mapper/reducer resource 
request from config files. When we have large config file, and large number of 
splits, it could take a long time.  

We saw AM could take hours to parse config when we have 200k+ splits, with a 
large config file (hundreds of kbs). 

We should do proper caching for pre-configured resource requests.

  was:
This is an issue could affect all the releases which includes YARN-6927. 

Basically, we use regex match repeatly when we read mapper/reducer resource 
request from config files. When we have large config file, and large number of 
splits, it could take a long time.  

We saw AM could take hours to parse config when we have 200k+ splits, with a 
large config file (hundreds of kbs). 

We should do proper caching for pre-configured resource requests.


> Improve performance of reading resource request for mapper/reducers from 
> config
> ---
>
> Key: MAPREDUCE-7309
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster
>Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0
>Reporter: Wangda Tan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309-004.patch, 
> MAPREDUCE-7309-005.patch, MAPREDUCE-7309.001.patch, MAPREDUCE-7309.002.patch
>
>
> This is an issue could affect all the releases which includes YARN-6927. 
> Basically, we use regex match repeatedly when we read mapper/reducer resource 
> request from config files. When we have large config file, and large number 
> of splits, it could take a long time.  
> We saw AM could take hours to parse config when we have 200k+ splits, with a 
> large config file (hundreds of kbs). 
> We should do proper caching for pre-configured resource requests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7266:
--
Fix Version/s: (was: 3.4.0)

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.3.0, 3.2.2
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079179#comment-17079179
 ] 

Szilard Nemeth commented on MAPREDUCE-7266:
---

OK, 3.3. and 3.2 backports went without conflict so I also committed to those 
branches. Resolving jira.

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.3.0, 3.2.2, 3.4.0
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7266:
--
Hadoop Flags: Reviewed
  Resolution: Fixed
  Status: Resolved  (was: Patch Available)

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.3.0, 3.2.2, 3.4.0
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7266:
--
Fix Version/s: 3.2.2
   3.3.0

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.3.0, 3.2.2, 3.4.0
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7266:
--
Fix Version/s: 3.4.0

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-09 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079177#comment-17079177
 ] 

Szilard Nemeth commented on MAPREDUCE-7266:
---

Hi [~sahuja],
Agreed, you don't need to bother with this ASF license warning.
Latest patch LGTM, committed to trunk.
Checking if this can be backported without conflict to 3.3/3.2

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: MAPREDUCE-7266.002.patch, YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-07 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077253#comment-17077253
 ] 

Szilard Nemeth edited comment on MAPREDUCE-7266 at 4/7/20, 1:46 PM:


Hi [~sahuja],
Thanks for working on this patch, good finding.
Patch makes sense. 
You don't need the cast here for type HistoryContext, as HistoryClientService 
receives a HistoryContext and JobHistory implements this interface.

{code:java}
  @VisibleForTesting
  protected HistoryClientService createHistoryClientService() {
return new HistoryClientService((HistoryContext)jobHistoryService,
this.jhsDTSecretManager);
  }
{code}

Otherwise, patch looks good to me.
Please upload a second patch, hopefully it will generate a green Jenkins build 
as well.


was (Author: snemeth):
Hi [~sahuja],
Patch makes sense. 
You don't need the cast here for type HistoryContext, as HistoryClientService 
receives a HistoryContext and JobHistory implements this interface.

{code:java}
  @VisibleForTesting
  protected HistoryClientService createHistoryClientService() {
return new HistoryClientService((HistoryContext)jobHistoryService,
this.jhsDTSecretManager);
  }
{code}

Otherwise, patch looks good to me.
Please upload a second patch, hopefully it will generate a green Jenkins build 
as well.

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7266) historyContext doesn't need to be a class attribute inside JobHistoryServer

2020-04-07 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077253#comment-17077253
 ] 

Szilard Nemeth commented on MAPREDUCE-7266:
---

Hi [~sahuja],
Patch makes sense. 
You don't need the cast here for type HistoryContext, as HistoryClientService 
receives a HistoryContext and JobHistory implements this interface.

{code:java}
  @VisibleForTesting
  protected HistoryClientService createHistoryClientService() {
return new HistoryClientService((HistoryContext)jobHistoryService,
this.jhsDTSecretManager);
  }
{code}

Otherwise, patch looks good to me.
Please upload a second patch, hopefully it will generate a green Jenkins build 
as well.

> historyContext doesn't need to be a class attribute inside JobHistoryServer
> ---
>
> Key: MAPREDUCE-7266
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7266
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: YARN-10075.001.patch
>
>
> "historyContext" class attribute at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  is assigned a cast of another class attribute - "jobHistoryService" - 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131,
>  however it does not need to be stored separately because it is only ever 
> used once in the clas, and that too as an argument while instantiating the 
> HistoryClientService class at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L155.
> Therefore, we could just delete the lines at 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L67
>  and 
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java#L131
>  completely and instantiate the HistoryClientService as follows:
> {code}
>   @VisibleForTesting
>   protected HistoryClientService createHistoryClientService() {
> return new HistoryClientService((HistoryContext)jobHistoryService, 
> this.jhsDTSecretManager);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7263) Remove obsolete validateTargetPath() from FrameworkUploader

2020-02-12 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7263:
--
Hadoop Flags: Reviewed

> Remove obsolete validateTargetPath() from FrameworkUploader
> ---
>
> Key: MAPREDUCE-7263
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7263
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 3.2.1
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7263.001.patch
>
>
> The following piece of code has been in the FrameworkUploader class for a 
> while:
> {code:java}
>   private void validateTargetPath() throws UploaderException {
> if (!target.startsWith("hdfs:/") &&
> !target.startsWith("file:/")) {
>   throw new UploaderException("Target path is not hdfs or local " + 
> target);
> }
>   }
> {code}
> As parts of MR and YARN has evolved, you can run YARN with OzoneFS as storage 
> for instance - so HDFS (or local) is not a requirement anymore. Also it does 
> not make sense to add other suitable filesystems when they're tested against 
> the MR framework, so I'm suggesting to remove the check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7263) Remove obsolete validateTargetPath() from FrameworkUploader

2020-02-12 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7263:
--
Fix Version/s: 3.3.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Remove obsolete validateTargetPath() from FrameworkUploader
> ---
>
> Key: MAPREDUCE-7263
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7263
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 3.2.1
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7263.001.patch
>
>
> The following piece of code has been in the FrameworkUploader class for a 
> while:
> {code:java}
>   private void validateTargetPath() throws UploaderException {
> if (!target.startsWith("hdfs:/") &&
> !target.startsWith("file:/")) {
>   throw new UploaderException("Target path is not hdfs or local " + 
> target);
> }
>   }
> {code}
> As parts of MR and YARN has evolved, you can run YARN with OzoneFS as storage 
> for instance - so HDFS (or local) is not a requirement anymore. Also it does 
> not make sense to add other suitable filesystems when they're tested against 
> the MR framework, so I'm suggesting to remove the check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7263) Remove obsolete validateTargetPath() from FrameworkUploader

2020-02-12 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035412#comment-17035412
 ] 

Szilard Nemeth commented on MAPREDUCE-7263:
---

Thanks [~mhudaky] for your patch, LGTM.

Committed to trunk.

Resolving jira.

> Remove obsolete validateTargetPath() from FrameworkUploader
> ---
>
> Key: MAPREDUCE-7263
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7263
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv2
>Affects Versions: 3.2.1
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Major
> Attachments: MAPREDUCE-7263.001.patch
>
>
> The following piece of code has been in the FrameworkUploader class for a 
> while:
> {code:java}
>   private void validateTargetPath() throws UploaderException {
> if (!target.startsWith("hdfs:/") &&
> !target.startsWith("file:/")) {
>   throw new UploaderException("Target path is not hdfs or local " + 
> target);
> }
>   }
> {code}
> As parts of MR and YARN has evolved, you can run YARN with OzoneFS as storage 
> for instance - so HDFS (or local) is not a requirement anymore. Also it does 
> not make sense to add other suitable filesystems when they're tested against 
> the MR framework, so I'm suggesting to remove the check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7260) Cross origin request support for Job history server web UI

2020-01-29 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7260:
--
Fix Version/s: 3.2.2
   3.3.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> Cross origin request support for Job history server web UI
> --
>
> Key: MAPREDUCE-7260
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7260
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Critical
> Fix For: 3.3.0, 3.2.2
>
> Attachments: MAPREDUCE-7260.001.patch, MAPREDUCE-7260.002.patch, 
> MAPREDUCE-7260.branch-3.2.001.patch
>
>
> The major web UIs in YARN support configuring the header, but somehow for the 
> JHS web UI there wasn't any use case. We need it in YARN-10029, so let's 
> implement it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7260) Cross origin request support for Job history server web UI

2020-01-29 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7260:
--
Hadoop Flags: Reviewed

> Cross origin request support for Job history server web UI
> --
>
> Key: MAPREDUCE-7260
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7260
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Critical
> Fix For: 3.3.0, 3.2.2
>
> Attachments: MAPREDUCE-7260.001.patch, MAPREDUCE-7260.002.patch, 
> MAPREDUCE-7260.branch-3.2.001.patch
>
>
> The major web UIs in YARN support configuring the header, but somehow for the 
> JHS web UI there wasn't any use case. We need it in YARN-10029, so let's 
> implement it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7260) Cross origin request support for Job history server web UI

2020-01-29 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17025882#comment-17025882
 ] 

Szilard Nemeth commented on MAPREDUCE-7260:
---

Hi [~adam.antal],
No need to upload it again.
Just checked your patches, LGTM, committed to trunk and branch-3.2.
Resolving jira.

> Cross origin request support for Job history server web UI
> --
>
> Key: MAPREDUCE-7260
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7260
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Critical
> Attachments: MAPREDUCE-7260.001.patch, MAPREDUCE-7260.002.patch, 
> MAPREDUCE-7260.branch-3.2.001.patch
>
>
> The major web UIs in YARN support configuring the header, but somehow for the 
> JHS web UI there wasn't any use case. We need it in YARN-10029, so let's 
> implement it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes

2019-10-18 Thread Szilard Nemeth (Jira)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954607#comment-16954607
 ] 

Szilard Nemeth commented on MAPREDUCE-6441:
---

Thanks [~pbacsko]!
Committed the patch for branch-3.1

> Improve temporary directory name generation in LocalDistributedCacheManager 
> for concurrent processes
> 
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Fix For: 3.2.0, 3.1.4
>
> Attachments: HADOOP-10924.02.patch, 
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441-branch-3.1.001.patch, 
> MAPREDUCE-6441.004.patch, MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, 
> MAPREDUCE-6441.008.patch, MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, 
> MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400:  INFO - 14/08/01 13:47:22 ERROR tool.ImportTool: 
> Encountered IOException running import job: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
> overwrite non empty destination directory 
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of 
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class: 
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and 
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes

2019-10-18 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-6441:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Improve temporary directory name generation in LocalDistributedCacheManager 
> for concurrent processes
> 
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Fix For: 3.1.4, 3.2.0
>
> Attachments: HADOOP-10924.02.patch, 
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441-branch-3.1.001.patch, 
> MAPREDUCE-6441.004.patch, MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, 
> MAPREDUCE-6441.008.patch, MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, 
> MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400:  INFO - 14/08/01 13:47:22 ERROR tool.ImportTool: 
> Encountered IOException running import job: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
> overwrite non empty destination directory 
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of 
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class: 
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and 
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6441) Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes

2019-10-18 Thread Szilard Nemeth (Jira)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-6441:
--
Fix Version/s: 3.1.4

> Improve temporary directory name generation in LocalDistributedCacheManager 
> for concurrent processes
> 
>
> Key: MAPREDUCE-6441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6441
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: William Watson
>Assignee: Haibo Chen
>Priority: Major
> Fix For: 3.2.0, 3.1.4
>
> Attachments: HADOOP-10924.02.patch, 
> HADOOP-10924.03.jobid-plus-uuid.patch, MAPREDUCE-6441-branch-3.1.001.patch, 
> MAPREDUCE-6441.004.patch, MAPREDUCE-6441.005.patch, MAPREDUCE-6441.006.patch, 
> MAPREDUCE-6441.008.patch, MAPREDUCE-6441.009.patch, MAPREDUCE-6441.010.patch, 
> MAPREDUCE-6441.011.patch
>
>
> Kicking off many sqoop processes in different threads results in:
> {code}
> 2014-08-01 13:47:24 -0400:  INFO - 14/08/01 13:47:22 ERROR tool.ImportTool: 
> Encountered IOException running import job: java.io.IOException: 
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot 
> overwrite non empty destination directory 
> /tmp/hadoop-hadoop/mapred/local/1406915233073
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:149)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:163)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> java.security.AccessController.doPrivileged(Native Method)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> javax.security.auth.Subject.doAs(Subject.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:645)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> 2014-08-01 13:47:24 -0400:  INFO -at 
> org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> {code}
> If two are kicked off in the same second. The issue is the following lines of 
> code in the org.apache.hadoop.mapred.LocalDistributedCacheManager class: 
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
>new AtomicLong(System.currentTimeMillis());
> {code}
> and 
> {code}
> Long.toString(uniqueNumberGenerator.incrementAndGet())),
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7230) TestHSWebApp.testLogsViewSingle fails

2019-08-15 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7230:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
   3.2.1
   3.3.0
   Status: Resolved  (was: Patch Available)

> TestHSWebApp.testLogsViewSingle fails
> -
>
> Key: MAPREDUCE-7230
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7230
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: MAPREDUCE-7230-001.patch
>
>
> TestHSWebApp.testLogsViewSingle fails.
> {code}
> [ERROR] 
> testLogsViewSingle(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp)  
> Time elapsed: 0.294 s  <<< FAILURE!
> Argument(s) are different! Wanted:
> printWriter.write(
> "Logs not available for container_10_0001_01_01. Aggregation may not 
> be complete, Check back later or try the nodemanager at localhost:1234"
> );
> -> at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewSingle(TestHSWebApp.java:234)
> Actual invocations have different arguments:
> printWriter.print(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.echoWithoutEscapeHtml(TextView.java:62)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at java.io.PrintWriter.print(PrintWriter.java:617)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>",
> 0,
> 90
> );
> -> at java.io.PrintWriter.write(PrintWriter.java:473)
> printWriter.println(
> 
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.putWithoutEscapeHtml(TextView.java:81)
> printWriter.print(
> " );
> -> at 
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl.printStartTag(HamletImpl.java:273)
> printWriter.write(
> " );
> -> at java.io.PrintWriter.print(PrintWriter.java:603)
> printWriter.write(
> " 0,
> 5
> );
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7230) TestHSWebApp.testLogsViewSingle fails

2019-08-15 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908160#comment-16908160
 ] 

Szilard Nemeth commented on MAPREDUCE-7230:
---

Hi [~Prabhu Joseph]!
Pushed patch to trunk, bracnch-3.2 and branch-3.1!
Thanks for your contribution!

> TestHSWebApp.testLogsViewSingle fails
> -
>
> Key: MAPREDUCE-7230
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7230
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: MAPREDUCE-7230-001.patch
>
>
> TestHSWebApp.testLogsViewSingle fails.
> {code}
> [ERROR] 
> testLogsViewSingle(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp)  
> Time elapsed: 0.294 s  <<< FAILURE!
> Argument(s) are different! Wanted:
> printWriter.write(
> "Logs not available for container_10_0001_01_01. Aggregation may not 
> be complete, Check back later or try the nodemanager at localhost:1234"
> );
> -> at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewSingle(TestHSWebApp.java:234)
> Actual invocations have different arguments:
> printWriter.print(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.echoWithoutEscapeHtml(TextView.java:62)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at java.io.PrintWriter.print(PrintWriter.java:617)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>",
> 0,
> 90
> );
> -> at java.io.PrintWriter.write(PrintWriter.java:473)
> printWriter.println(
> 
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.putWithoutEscapeHtml(TextView.java:81)
> printWriter.print(
> " );
> -> at 
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl.printStartTag(HamletImpl.java:273)
> printWriter.write(
> " );
> -> at java.io.PrintWriter.print(PrintWriter.java:603)
> printWriter.write(
> " 0,
> 5
> );
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7230) TestHSWebApp.testLogsViewSingle fails

2019-08-14 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907434#comment-16907434
 ] 

Szilard Nemeth commented on MAPREDUCE-7230:
---

Hi [~Prabhu Joseph]!
Fix looks good to me!
One question: As YARN-9451 was merged to trunk and backported to 3.2 / 3.1, 
should we do the same with this fix, right?

Thanks!

> TestHSWebApp.testLogsViewSingle fails
> -
>
> Key: MAPREDUCE-7230
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7230
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, test
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: MAPREDUCE-7230-001.patch
>
>
> TestHSWebApp.testLogsViewSingle fails.
> {code}
> [ERROR] 
> testLogsViewSingle(org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp)  
> Time elapsed: 0.294 s  <<< FAILURE!
> Argument(s) are different! Wanted:
> printWriter.write(
> "Logs not available for container_10_0001_01_01. Aggregation may not 
> be complete, Check back later or try the nodemanager at localhost:1234"
> );
> -> at 
> org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp.testLogsViewSingle(TestHSWebApp.java:234)
> Actual invocations have different arguments:
> printWriter.print(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.echoWithoutEscapeHtml(TextView.java:62)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>"
> );
> -> at java.io.PrintWriter.print(PrintWriter.java:617)
> printWriter.write(
> " "http://www.w3.org/TR/html4/strict.dtd;>",
> 0,
> 90
> );
> -> at java.io.PrintWriter.write(PrintWriter.java:473)
> printWriter.println(
> 
> );
> -> at 
> org.apache.hadoop.yarn.webapp.view.TextView.putWithoutEscapeHtml(TextView.java:81)
> printWriter.print(
> " );
> -> at 
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl.printStartTag(HamletImpl.java:273)
> printWriter.write(
> " );
> -> at java.io.PrintWriter.print(PrintWriter.java:603)
> printWriter.write(
> " 0,
> 5
> );
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-12 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7197:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
   Status: Resolved  (was: Patch Available)

> Fix order of actual and expected expression in assert statements
> 
>
> Key: MAPREDUCE-7197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7197
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch, 
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch, 
> MAPREDUCE-7197.005.patch, MAPREDUCE-7197.006.patch, 
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives 
> misleading message when test case fails. Attached file has some of the places 
> where it is placed wrongly.
> {code:java}
> [ERROR] 
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
>   Time elapsed: 3.385 s  <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but 
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be 
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-12 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905114#comment-16905114
 ] 

Szilard Nemeth commented on MAPREDUCE-7197:
---

Hi [~adam.antal]!
Thanks for this patch, committed to trunk!

> Fix order of actual and expected expression in assert statements
> 
>
> Key: MAPREDUCE-7197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7197
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch, 
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch, 
> MAPREDUCE-7197.005.patch, MAPREDUCE-7197.006.patch, 
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives 
> misleading message when test case fails. Attached file has some of the places 
> where it is placed wrongly.
> {code:java}
> [ERROR] 
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
>   Time elapsed: 3.385 s  <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but 
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be 
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-09 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903970#comment-16903970
 ] 

Szilard Nemeth commented on MAPREDUCE-7197:
---

Hi [~adam.antal]!
We can be more tolerant and don't bother with the ASF license issue and the 
checkstyle issue, so I'm giving +1 for this.
Will commit next week if no one has an objection until then!

> Fix order of actual and expected expression in assert statements
> 
>
> Key: MAPREDUCE-7197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7197
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch, 
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch, 
> MAPREDUCE-7197.005.patch, MAPREDUCE-7197.006.patch, 
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives 
> misleading message when test case fails. Attached file has some of the places 
> where it is placed wrongly.
> {code:java}
> [ERROR] 
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
>   Time elapsed: 3.385 s  <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but 
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be 
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-05 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899900#comment-16899900
 ] 

Szilard Nemeth commented on MAPREDUCE-7197:
---

Hi [~adam.antal]! 
Thanks for this patch, it's quite extensive and precise.
However, I have some comments:

1. In 
org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler#testAMStartedEvent:
 
There are 2 code blocks you modified: 

a. 

{code:java}
assertThat(100).isEqualTo(mi.getJobIndexInfo().getSubmitTime());
assertThat(200).isEqualTo(mi.getJobIndexInfo().getJobStartTime());
assertThat(100).isEqualTo(mi.getJobSummary().getJobSubmitTime());
assertThat(200).isEqualTo(mi.getJobSummary().getJobLaunchTime());
{code}


b. 

{code:java}
assertThat(100).isEqualTo(mi.getJobIndexInfo().getSubmitTime());
assertThat(200).isEqualTo(mi.getJobIndexInfo().getJobStartTime());
assertThat(100).isEqualTo(mi.getJobSummary().getJobSubmitTime());
assertThat(200).isEqualTo(mi.getJobSummary().getJobLaunchTime());
{code}


Even though the original code had the expected and actual values interchanged, 
you should change the expected values with actual for these, for example: 

{code:java}
assertThat(mi.getJobIndexInfo().getSubmitTime()).isEqualTo(100); --> Expected 
value was 100.
{code}




2. In 
org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators#coreTestEstimator: 
You removed one meaningful assertion with your patch: 
You have: 

{code:java}
assertThat(speculator.getProportionRunningTasksSpeculatable())
.isCloseTo(0.1, offset(0.1));
{code}


And we had: 

{code:java}
Assert.assertEquals(speculator.getProportionRunningTasksSpeculatable(),
0.1, 0.1);
Assert.assertEquals(speculator.getProportionTotalTasksSpeculatable(),
0.001, 0.1);

{code}

, so the assertion for speculator.getProportionTotalTasksSpeculatable() is 
missing.


3. In 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt#testContainerCleanedWhileRunning:
 
You have this change: 


{code:java}
assertThat(taImpl.getState())
.withFailMessage("Task attempt is not in running state")
.isEqualTo(TaskAttemptState.RUNNING);
{code}


I'd prefer to move the call to isEqualTo above withFailMessage. 
You have similar call structures throughout this class, please also change all 
of them!

4. In 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt#testFetchFailureAttemptFinishTime:
 
This assertion is weird, the message is not in line with the state type, please 
fix it! : 

{code:java}
assertThat(taImpl.getState())
.withFailMessage("Task attempt is not in Too Many Fetch Failure state")
.isEqualTo(TaskAttemptState.FAILED);
{code}


5. In 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt#testContainerKillAfterAssigned:
 
There's a typo (not introduced by you) in this assertion message: 

{code:java}
assertThat(taImpl.getInternalState())
.withFailMessage("Task attempt is not in assinged state")
.isEqualTo(TaskAttemptStateInternal.ASSIGNED);
{code}


6. In 
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt#testTaskAttemptDiagnosticEventOnFinishing:
 
I like that you fixed some assertion message here: 

{code:java}
assertThat(taImpl.getState())
.withFailMessage("Task attempt is not in succeeded state")
.isEqualTo(TaskAttemptState.SUCCEEDED); 
{code}


However, please use the name of the state with uppercase letters. You have one 
more occurrence of this in the same method.

7. In org.apache.hadoop.mapred.TestJobConf: 
I can see some updates regarding enum fields (asserting them directly without 
using .name() ) but I can't see assertEquals rewritten to assertThat for a 
bunch of occurrences in the class, please fix this!

8. In org.apache.hadoop.mapreduce.task.reduce.TestMergeManager: 
Your 2 changes in this class are using assertNull. Can you please use 
assertThat + isNull instead? 

You have an occurrence of this in the following palces as well:

{code:java}
- org.apache.hadoop.mapred.TestTaskLog#testTaskLogWithoutTaskLogDir 
- In class org.apache.hadoop.mapreduce.security.TestTokenCache
- org.apache.hadoop.mapred.uploader.TestFrameworkUploader#testHelp
{code}


9. In 
org.apache.hadoop.mapreduce.v2.hs.TestJHSDelegationTokenSecretManager#stopAndCleanSecretManager:
 
You have this assertion: 

{code:java}
assertThat(mgr.getAllTokens().size())
.withFailMessage("Secret manager should not contain keys").isZero();
{code}


The original assertion was: 

{code:java}
Assert.assertEquals("Secret manager should not contain tokens",
mgr.getAllTokens().size(), 0);
{code}


The assertion message should have the word "tokens" instead of "keys", I think.

10. In 
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities#testCompletedJob: 
You rewritten one assertEquals to assertTrue. Can you use assertThat instead? 

11. In

[jira] [Updated] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-02 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7197:
--
Attachment: MAPREDUCE-7197.004.patch

> Fix order of actual and expected expression in assert statements
> 
>
> Key: MAPREDUCE-7197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7197
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch, 
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, MAPREDUCE-7197.004.patch, 
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives 
> misleading message when test case fails. Attached file has some of the places 
> where it is placed wrongly.
> {code:java}
> [ERROR] 
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
>   Time elapsed: 3.385 s  <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but 
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be 
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7197) Fix order of actual and expected expression in assert statements

2019-08-02 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898823#comment-16898823
 ] 

Szilard Nemeth commented on MAPREDUCE-7197:
---

Hi [~adam.antal]!
I'm about to reupload your latest patch.
Could you please verify the checkstyle and asflicense issues? I can see we had 
some of those, but the results are not available anymore.

Thanks!

> Fix order of actual and expected expression in assert statements
> 
>
> Key: MAPREDUCE-7197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7197
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7197.001.patch, MAPREDUCE-7197.002.patch, 
> MAPREDUCE-7197.003.patch, MAPREDUCE-7197.004.patch, 
> old-approx-from-HADOOP-12693
>
>
> Fix order of actual and expected expression in assert statements which gives 
> misleading message when test case fails. Attached file has some of the places 
> where it is placed wrongly.
> {code:java}
> [ERROR] 
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
>   Time elapsed: 3.385 s  <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but 
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be 
> used for new test cases which avoids such mistakes.
> This is a follow-up Jira on the fix for the MR project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7225) Fix broken current folder expansion during MR job start

2019-08-01 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898112#comment-16898112
 ] 

Szilard Nemeth commented on MAPREDUCE-7225:
---

Commited branch-3.2 and branch-3.1 patches as well.
Thanks [~pbacsko] for the patch and [~adam.antal] and [~wilfreds] for the 
review!

> Fix broken current folder expansion during MR job start
> ---
>
> Key: MAPREDUCE-7225
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7225
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 2.9.0, 3.0.3
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7225-001.patch, MAPREDUCE-7225-002.patch, 
> MAPREDUCE-7225-002.patch, MAPREDUCE-7225-003.patch, 
> MAPREDUCE-7225.branch-3.1.001.patch, MAPREDUCE-7225.branch-3.2.001.patch
>
>
> Starting a sleep job giving "." as files that should be localized is working 
> fine up until 2.9.0, but after that the user is given an 
> IllegalArgumentException. This change is a side-effect of HADOOP-12747 where 
> {{GenericOptionsParser#validateFiles}} function got modified.
> Can be reproduced by starting a sleep job with "-files ." given as extra 
> parameter. Log:
> {noformat}
> sudo -u hdfs hadoop jar hadoop-mapreduce-client-jobclient-3.0.0.jar sleep 
> -files . -m 1 -r 1 -rt 2000 -mt 2000
> WARNING: Use "yarn jar" to launch YARN applications.
> 19/07/17 08:13:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over 
> to rm21
> 19/07/17 08:13:26 INFO mapreduce.JobResourceUploader: Disabling Erasure 
> Coding for path: /user/hdfs/.staging/job_1563349475208_0017
> 19/07/17 08:13:26 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /user/hdfs/.staging/job_1563349475208_0017
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
>   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)
>   at org.apache.hadoop.fs.Path.(Path.java:180)
>   at org.apache.hadoop.fs.Path.(Path.java:125)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:686)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:262)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:203)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:131)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
>   at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at 
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:139)
>   at 
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7225) Fix broken current folder expansion during MR job start

2019-08-01 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897977#comment-16897977
 ] 

Szilard Nemeth commented on MAPREDUCE-7225:
---

Hi [~pbacsko]!
Thanks for this fix!
Read through the discussion and reviews and checked your patch. Latest patch 
looks good, +1. Committing it to trunk.
Please also provide branch-3.2 and branch-3.1 patches as well.

> Fix broken current folder expansion during MR job start
> ---
>
> Key: MAPREDUCE-7225
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7225
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 2.9.0, 3.0.3
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7225-001.patch, MAPREDUCE-7225-002.patch, 
> MAPREDUCE-7225-002.patch, MAPREDUCE-7225-003.patch
>
>
> Starting a sleep job giving "." as files that should be localized is working 
> fine up until 2.9.0, but after that the user is given an 
> IllegalArgumentException. This change is a side-effect of HADOOP-12747 where 
> {{GenericOptionsParser#validateFiles}} function got modified.
> Can be reproduced by starting a sleep job with "-files ." given as extra 
> parameter. Log:
> {noformat}
> sudo -u hdfs hadoop jar hadoop-mapreduce-client-jobclient-3.0.0.jar sleep 
> -files . -m 1 -r 1 -rt 2000 -mt 2000
> WARNING: Use "yarn jar" to launch YARN applications.
> 19/07/17 08:13:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over 
> to rm21
> 19/07/17 08:13:26 INFO mapreduce.JobResourceUploader: Disabling Erasure 
> Coding for path: /user/hdfs/.staging/job_1563349475208_0017
> 19/07/17 08:13:26 INFO mapreduce.JobSubmitter: Cleaning up the staging area 
> /user/hdfs/.staging/job_1563349475208_0017
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
>   at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)
>   at org.apache.hadoop.fs.Path.(Path.java:180)
>   at org.apache.hadoop.fs.Path.(Path.java:125)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:686)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:262)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:203)
>   at 
> org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:131)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
>   at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
>   at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
>   at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
>   at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at 
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:139)
>   at 
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-28 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804323#comment-16804323
 ] 

Szilard Nemeth commented on MAPREDUCE-7190:
---

Hi [~tangzhankun], [~sunilg]!

Can you please check and commit this? 

This is pretty important to us to proceed further with our testing fixes.

Thanks!

> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7190.001.patch, MAPREDUCE-7190.002.patch
>
>
> In our testing framework, we use SleepJobs for test scheduling and to check 
> different resource types are assigned properly to containers. Sometimes we 
> start multiple jobs in parallel, and if one fails, we have to track which one 
> was that. It is not too hard to find out that information, but would have 
> been much more convenient if it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-11 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789554#comment-16789554
 ] 

Szilard Nemeth commented on MAPREDUCE-7190:
---

Hi [~adam.antal]!

Thanks for the updated patch, giving a +1 for that!

I think you are right about not caring about the name "Sleep job - Sleep job".

Can you please check the UT failure (testMRNewTimelineServiceEventHandling)? I 
saw both patches generated UT issues in the same class, probably it's related.

Thanks!

> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7190.001.patch, MAPREDUCE-7190.002.patch
>
>
> In our testing framework, we use SleepJobs for test scheduling and to check 
> different resource types are assigned properly to containers. Sometimes we 
> start multiple jobs in parallel, and if one fails, we have to track which one 
> was that. It is not too hard to find out that information, but would have 
> been much more convenient if it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-08 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788078#comment-16788078
 ] 

Szilard Nemeth edited comment on MAPREDUCE-7190 at 3/8/19 5:13 PM:
---

Thanks [~adam.antal] for this patch, this is quite valuable and nice work!

One comment: 
 For the new createJob method, can't you use SLEEP_JOB_NAME as the default name 
value instead of null? 
 Then in the other createJob method (that has String name as the last 
parameter), instead of the null and empty-string check, you could check that if 
the name not equals SLEEP_JOB_NAME, then you need to set the name with the 
SLEEP_JOB_NAME prefix: 
{code:java}
job.setJobName(SLEEP_JOB_NAME + " - " + name);{code}
I think this way, the code would be much cleaner.

Please fix the checkstyle issues and check if the UT failure is related.

Thanks!


was (Author: snemeth):
Thanks [~adam.antal] for this patch, this is quite valuable and nice work!

One comment: 
For the new createJob method, can't you use SLEEP_JOB_NAME as the default name 
value instead of null? 
Then in the other createJob method (that has String name as the last 
parameter), instead of the null and empty-string check, you could check that if 
the name not equals SLEEP_JOB_NAME, then you need to set the name with the 
SLEEP_JOB_NAME prefix: 
job.setJobName(SLEEP_JOB_NAME + " - " + name);

I think this way, the code would be much cleaner.

Please fix the checkstyle issues and check if the UT failure is related.

Thanks!

> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7190.001.patch
>
>
> In our testing framework, we use SleepJobs for test scheduling and to check 
> different resource types are assigned properly to containers. Sometimes we 
> start multiple jobs in parallel, and if one fails, we have to track which one 
> was that. It is not too hard to find out that information, but would have 
> been much more convenient if it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-08 Thread Szilard Nemeth (JIRA)



[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788078#comment-16788078
 ] 

Szilard Nemeth commented on MAPREDUCE-7190:
---

Thanks [~adam.antal] for this patch, this is quite valuable and nice work!

One comment: 
For the new createJob method, can't you use SLEEP_JOB_NAME as the default name 
value instead of null? 
Then in the other createJob method (that has String name as the last 
parameter), instead of the null and empty-string check, you could check that if 
the name not equals SLEEP_JOB_NAME, then you need to set the name with the 
SLEEP_JOB_NAME prefix: 
job.setJobName(SLEEP_JOB_NAME + " - " + name);

I think this way, the code would be much cleaner.

Please fix the checkstyle issues and check if the UT failure is related.

Thanks!

> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Attachments: MAPREDUCE-7190.001.patch
>
>
> In our testing framework, we use SleepJobs for test scheduling and to check 
> different resource types are assigned properly to containers. Sometimes we 
> start multiple jobs in parallel, and if one fails, we have to track which one 
> was that. It is not too hard to find out that information, but would have 
> been much more convenient if it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-05 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7190:
--
Description: 
In our testing framework, we use SleepJobs for test scheduling and to check 
different resource types are assigned properly to containers. Sometimes we 
start multiple jobs in parallel, and if one fails, we have to track which one 
was that. It is not too hard to find out that information, but would have been 
much more convenient if it was written in the name of the job.

Proposal: additional parameter to SleepJob, so that its name can be changed 
from default "Sleep job." to something like "Sleep job - _additional info_".

  was:
In our testing framework we use SleepJobs for testing scheduling, and to check 
different resource types are assigned properly to containers. Sometimes we 
start multiple jobs in parallel, and if one fails, we have to track which one 
was that. It is not too hard to find out that information, but would have been 
much more convenient, if we it was written in the name of the job.

Proposal: additional parameter to SleepJob, so that its name can be changed 
from default "Sleep job." to something like "Sleep job - _additional info_".


> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> In our testing framework, we use SleepJobs for test scheduling and to check 
> different resource types are assigned properly to containers. Sometimes we 
> start multiple jobs in parallel, and if one fails, we have to track which one 
> was that. It is not too hard to find out that information, but would have 
> been much more convenient if it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7190) Add SleepJob additional parameter to make parallel runs distinguishable

2019-03-05 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7190:
--
Summary: Add SleepJob additional parameter to make parallel runs 
distinguishable  (was: Add SleepJob additional parameter to make paralel runs 
distinguishable)

> Add SleepJob additional parameter to make parallel runs distinguishable
> ---
>
> Key: MAPREDUCE-7190
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7190
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
>
> In our testing framework we use SleepJobs for testing scheduling, and to 
> check different resource types are assigned properly to containers. Sometimes 
> we start multiple jobs in parallel, and if one fails, we have to track which 
> one was that. It is not too hard to find out that information, but would have 
> been much more convenient, if we it was written in the name of the job.
> Proposal: additional parameter to SleepJob, so that its name can be changed 
> from default "Sleep job." to something like "Sleep job - _additional info_".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7161) Replace all MockRM submit method definitions with builder

2018-11-24 Thread Szilard Nemeth (JIRA)

Szilard Nemeth created MAPREDUCE-7161:
-

 Summary: Replace all MockRM submit method definitions with builder
 Key: MAPREDUCE-7161
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7161
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth


MockRM has 31 definitions of submitApp, most of them having more than 
acceptable number of parameters, ranging from 2 to even 22 parameters, which 
makes the code completely unreadable.
On top of unreadability, it's very hard to follow what RmApp will be produced 
for tests as they often pass a lot of empty / null values as parameters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7113) Typos in test names in TestTaskAttempt: "testAppDiognostic"

2018-06-19 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7113:
--
Priority: Minor  (was: Major)

> Typos in test names in TestTaskAttempt: "testAppDiognostic"
> ---
>
> Key: MAPREDUCE-7113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7113
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: MAPREDUCE-7113.001.patch
>
>
> These two methods need to be renamed: 
>  * testAppDiognosticEventOnUnassignedTask
>  * testAppDiognosticEventOnNewTask



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7113) Typos in test names in TestTaskAttempt: "testAppDiognostic"

2018-06-19 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7113:
--
Status: Patch Available  (was: In Progress)

> Typos in test names in TestTaskAttempt: "testAppDiognostic"
> ---
>
> Key: MAPREDUCE-7113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7113
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: MAPREDUCE-7113.001.patch
>
>
> These two methods need to be renamed: 
>  * testAppDiognosticEventOnUnassignedTask
>  * testAppDiognosticEventOnNewTask



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Work started] (MAPREDUCE-7113) Typos in test names in TestTaskAttempt: "testAppDiognostic"

2018-06-19 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on MAPREDUCE-7113 started by Szilard Nemeth.
-
> Typos in test names in TestTaskAttempt: "testAppDiognostic"
> ---
>
> Key: MAPREDUCE-7113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7113
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: MAPREDUCE-7113.001.patch
>
>
> These two methods need to be renamed: 
>  * testAppDiognosticEventOnUnassignedTask
>  * testAppDiognosticEventOnNewTask



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7113) Typos in test names in TestTaskAttempt: "testAppDiognostic"

2018-06-19 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth updated MAPREDUCE-7113:
--
Attachment: MAPREDUCE-7113.001.patch

> Typos in test names in TestTaskAttempt: "testAppDiognostic"
> ---
>
> Key: MAPREDUCE-7113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7113
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: MAPREDUCE-7113.001.patch
>
>
> These two methods need to be renamed: 
>  * testAppDiognosticEventOnUnassignedTask
>  * testAppDiognosticEventOnNewTask



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Moved] (MAPREDUCE-7113) Typos in test names in TestTaskAttempt: "testAppDiognostic"

2018-06-19 Thread Szilard Nemeth (JIRA)



 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szilard Nemeth moved YARN-8439 to MAPREDUCE-7113:
-

Key: MAPREDUCE-7113  (was: YARN-8439)
Project: Hadoop Map/Reduce  (was: Hadoop YARN)

> Typos in test names in TestTaskAttempt: "testAppDiognostic"
> ---
>
> Key: MAPREDUCE-7113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7113
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
>
> These two methods need to be renamed: 
>  * testAppDiognosticEventOnUnassignedTask
>  * testAppDiognosticEventOnNewTask



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

53 matches

Mail list logo