[jira] [Updated] (MAPREDUCE-7167) Extra LF ("\n") pushed directly to storage
[
https://issues.apache.org/jira/browse/MAPREDUCE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7167:
Target Version/s: 3.2.5 (was: 3.2.4)
> Extra LF ("\n") pushed directly to storage
> --
>
> Key: MAPREDUCE-7167
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7167
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Saurabh
>Assignee: Saurabh
>Priority: Major
> Attachments: image-2018-11-28-19-23-52-972.png,
> image-2018-11-29-14-53-58-176.png, image-2018-11-29-14-54-28-254.png,
> nremoved.txt, nremoved.txt, patch1128.patch, patch1128.patch,
> patch1128trunk.patch, withn.txt, withn.txt
>
>
> JsonEncoder already adds the necessary newline after writing each object as
> per [this|
> [https://github.com/apache/avro/blob/39ec1a3f0addfce06869f705f7a17c03d538fe16/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java#L77]
> ] so this patch removes the "out.writeBytes("\n");". As the encoder is
> buffered, the out.writeBytes can cause JSON errors in the output stream as it
> directly writes to the output stream, hence it must be removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7167) Extra LF ("\n") pushed directly to storage
[
https://issues.apache.org/jira/browse/MAPREDUCE-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565899#comment-17565899
]
Masatake Iwasaki commented on MAPREDUCE-7167:
-
update the targets to 3.2.5 for preparing 3.2.4 release.
> Extra LF ("\n") pushed directly to storage
> --
>
> Key: MAPREDUCE-7167
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7167
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Saurabh
>Assignee: Saurabh
>Priority: Major
> Attachments: image-2018-11-28-19-23-52-972.png,
> image-2018-11-29-14-53-58-176.png, image-2018-11-29-14-54-28-254.png,
> nremoved.txt, nremoved.txt, patch1128.patch, patch1128.patch,
> patch1128trunk.patch, withn.txt, withn.txt
>
>
> JsonEncoder already adds the necessary newline after writing each object as
> per [this|
> [https://github.com/apache/avro/blob/39ec1a3f0addfce06869f705f7a17c03d538fe16/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java#L77]
> ] so this patch removes the "out.writeBytes("\n");". As the encoder is
> buffered, the out.writeBytes can cause JSON errors in the output stream as it
> directly writes to the output stream, hence it must be removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close
[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7369:
Fix Version/s: 3.4.0
3.3.4
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.4
>
> Attachments: MAPREDUCE-7369.001.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7373) Building MapReduce NativeTask fails on Fedora 34+
[
https://issues.apache.org/jira/browse/MAPREDUCE-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved MAPREDUCE-7373.
-
Fix Version/s: 3.4.0
3.2.4
3.3.3
Hadoop Flags: Reviewed
Resolution: Fixed
> Building MapReduce NativeTask fails on Fedora 34+
> -
>
> Key: MAPREDUCE-7373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7373
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build, nativetask
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.3
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Fedora 34 adopted GCC 11, in which C++17 features are enabled by default.
> https://gcc.gnu.org/projects/cxx-status.html#cxx17
> Building MapReduce NativeTask with it leads to the following error.
> (I found it on branch-3.2, but it's supposed to be the same as trunk)
> {code}
> $ mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
> ...
> [WARNING] In file included from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.h:30,
> [WARNING] from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/handler/MCollectorOutputHandler.cc:24:
> [WARNING]
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucket.h:127:36:
> error: ISO C++17 does not allow dynamic exception specifications
> [WARNING] 127 | void spill(IFileWriter * writer) throw (IOException,
> UnsupportException);
> [WARNING] |^
> [WARNING] make[2]: *** [CMakeFiles/nativetask_static.dir/build.make:160:
> CMakeFiles/nativetask_static.dir/main/native/src/handler/MCollectorOutputHandler.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/Makefile2:115:
> CMakeFiles/nativetask_static.dir/all] Error 2
> [WARNING] make: *** [Makefile:91: all] Error 2
> ...
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins .. SUCCESS [ 0.570
> s]
> [INFO] Apache Hadoop MapReduce NativeTask . FAILURE [ 11.016
> s]
> [INFO] Apache Hadoop MapReduce Uploader ... SKIPPED
> [INFO] Apache Hadoop MapReduce Examples ... SKIPPED
> [INFO] Apache Hadoop MapReduce SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming .. SKIPPED
> [INFO] Apache Hadoop Distributed Copy . SKIPPED
> [INFO] Apache Hadoop Archives . SKIPPED
> [INFO] Apache Hadoop Archive Logs . SKIPPED
> [INFO] Apache Hadoop Rumen SKIPPED
> [INFO] Apache Hadoop Gridmix .. SKIPPED
> [INFO] Apache Hadoop Data Join SKIPPED
> [INFO] Apache Hadoop Extras ... SKIPPED
> [INFO] Apache Hadoop Pipes SKIPPED
> [INFO] Apache Hadoop OpenStack support SKIPPED
> [INFO] Apache Hadoop Amazon Web Services support .. SKIPPED
> [INFO] Apache Hadoop Kafka Library support SKIPPED
> [INFO] Apache Hadoop Azure support SKIPPED
> [INFO] Apache Hadoop Aliyun OSS support ... SKIPPED
> [INFO] Apache Hadoop Client Aggregator SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED
> [INFO] Apache Hadoop Resource Estimator Service ... SKIPPED
> [INFO] Apache Hadoop Azure Data Lake support .. SKIPPED
> [INFO] Apache Hadoop Tools Dist ... SKIPPED
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Client API ... SKIPPED
> [INFO] Apache Hadoop Client Runtime ... SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants .. SKIPPED
> [INFO] Apache Hadoop Client Test Minicluster .. SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO] Apache Hadoop Client Modules ... SKIPPED
> [INFO] Apache Hadoop Cloud Storage SKIPPED
> [INFO] Apache Hadoop Cloud Storage Project SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
>
[jira] [Assigned] (MAPREDUCE-7373) Building MapReduce NativeTask fails on Fedora 34+
[
https://issues.apache.org/jira/browse/MAPREDUCE-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned MAPREDUCE-7373:
---
Assignee: Kengo Seki
> Building MapReduce NativeTask fails on Fedora 34+
> -
>
> Key: MAPREDUCE-7373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7373
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build, nativetask
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Fedora 34 adopted GCC 11, in which C++17 features are enabled by default.
> https://gcc.gnu.org/projects/cxx-status.html#cxx17
> Building MapReduce NativeTask with it leads to the following error.
> (I found it on branch-3.2, but it's supposed to be the same as trunk)
> {code}
> $ mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
> ...
> [WARNING] In file included from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.h:30,
> [WARNING] from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/handler/MCollectorOutputHandler.cc:24:
> [WARNING]
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucket.h:127:36:
> error: ISO C++17 does not allow dynamic exception specifications
> [WARNING] 127 | void spill(IFileWriter * writer) throw (IOException,
> UnsupportException);
> [WARNING] |^
> [WARNING] make[2]: *** [CMakeFiles/nativetask_static.dir/build.make:160:
> CMakeFiles/nativetask_static.dir/main/native/src/handler/MCollectorOutputHandler.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/Makefile2:115:
> CMakeFiles/nativetask_static.dir/all] Error 2
> [WARNING] make: *** [Makefile:91: all] Error 2
> ...
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins .. SUCCESS [ 0.570
> s]
> [INFO] Apache Hadoop MapReduce NativeTask . FAILURE [ 11.016
> s]
> [INFO] Apache Hadoop MapReduce Uploader ... SKIPPED
> [INFO] Apache Hadoop MapReduce Examples ... SKIPPED
> [INFO] Apache Hadoop MapReduce SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming .. SKIPPED
> [INFO] Apache Hadoop Distributed Copy . SKIPPED
> [INFO] Apache Hadoop Archives . SKIPPED
> [INFO] Apache Hadoop Archive Logs . SKIPPED
> [INFO] Apache Hadoop Rumen SKIPPED
> [INFO] Apache Hadoop Gridmix .. SKIPPED
> [INFO] Apache Hadoop Data Join SKIPPED
> [INFO] Apache Hadoop Extras ... SKIPPED
> [INFO] Apache Hadoop Pipes SKIPPED
> [INFO] Apache Hadoop OpenStack support SKIPPED
> [INFO] Apache Hadoop Amazon Web Services support .. SKIPPED
> [INFO] Apache Hadoop Kafka Library support SKIPPED
> [INFO] Apache Hadoop Azure support SKIPPED
> [INFO] Apache Hadoop Aliyun OSS support ... SKIPPED
> [INFO] Apache Hadoop Client Aggregator SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED
> [INFO] Apache Hadoop Resource Estimator Service ... SKIPPED
> [INFO] Apache Hadoop Azure Data Lake support .. SKIPPED
> [INFO] Apache Hadoop Tools Dist ... SKIPPED
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Client API ... SKIPPED
> [INFO] Apache Hadoop Client Runtime ... SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants .. SKIPPED
> [INFO] Apache Hadoop Client Test Minicluster .. SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO] Apache Hadoop Client Modules ... SKIPPED
> [INFO] Apache Hadoop Cloud Storage SKIPPED
> [INFO] Apache Hadoop Cloud Storage Project SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 04:10 min
> [INFO] Finished at:
[jira] [Commented] (MAPREDUCE-7301) Expose Mini MR Cluster attribute for testing
[ https://issues.apache.org/jira/browse/MAPREDUCE-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211834#comment-17211834 ] Masatake Iwasaki commented on MAPREDUCE-7301: - LGTM. If there is depending JIRA of HBase or Phoenix, linking it here would be help for understanding use cases. > Expose Mini MR Cluster attribute for testing > > > Key: MAPREDUCE-7301 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7301 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Reporter: Swaroopa Kadam >Assignee: Swaroopa Kadam >Priority: Minor > Attachments: MAPREDUCE-7301.patch > > > After MAPREDUCE-6521, applications using MiniMRYarnCluster need to explicitly > set the config "yarn.app.mapreduce.am.staging-dir" which ll include cluster > name in it. > > In order to get the cluster name, some getters on corresponding attributes > are needed to be available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7294) Only application master should upload resource to Yarn Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200339#comment-17200339 ] Masatake Iwasaki commented on MAPREDUCE-7294: - [~liuml07] the artifacts of 2.10.1 were already published. I'm updating site docs now. This should be targeted to 2.10.2. > Only application master should upload resource to Yarn Shared Cache > --- > > Key: MAPREDUCE-7294 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7294 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.4 >Reporter: zhenzhao wang >Assignee: zhenzhao wang >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2 > > > The design of yarn shared cache manager is only to allow application master > should upload the jar/files/resource. However, there was a bug in the code > since 2.9.0. Every node manager that take the job task will try to upload the > jar/resources. Let's say one job have 5000 tasks. Then there will be up to > 5000 NMs try to upload the jar. This is like DDOS and create a snowball > effect. It will end up with inavailability of yarn shared cache manager. It > wil cause time out in localization and lead to job failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7286) Fix findbugs warnings in hadoop-mapreduce-project on branch-2.10
[
https://issues.apache.org/jira/browse/MAPREDUCE-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved MAPREDUCE-7286.
-
Fix Version/s: 2.10.1
Resolution: Fixed
Cherry-picking MAPREDUCE-6881 fixed this.
> Fix findbugs warnings in hadoop-mapreduce-project on branch-2.10
>
>
> Key: MAPREDUCE-7286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.10.1
>
>
> {noformat}
> $ find . -name findbugsXml.xml | xargs -n 1
> /opt/findbugs-3.0.1/bin/convertXmlToText -longBugCodes
> M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
> dereference in org.apache.hadoop.examples.pi.Parser.parse(File, Map) due to
> return value of called method Dereferenced at Parser.java:[line 70]
> H P DM_BOXED_PRIMITIVE_FOR_COMPARE Bx: Primitive is boxed to call
> Long.compareTo(Long): use Long.compare(long, long) instead At
> JVMId.java:[line 101]
> M B ME_MUTABLE_ENUM_FIELD ME: org.apache.hadoop.mapred.Operation.jobACLNeeded
> field is public and mutable In Operation.java
> M B ME_MUTABLE_ENUM_FIELD ME: org.apache.hadoop.mapred.Operation.qACLNeeded
> field is public and mutable In Operation.java
> M D UC_USELESS_OBJECT UC: Useless object stored in variable paths of method
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.moveToDone()
> At HistoryFileManager.java:[line 416]
> M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
> dereference in new
> org.apache.hadoop.mapred.LocalContainerLauncher(AppContext,
> TaskUmbilicalProtocol, ClassLoader) due to return value of called method
> Dereferenced at LocalContainerLauncher.java:[line 124]
> M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
> dereference in
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.relocalize() due
> to return value of called method Dereferenced at
> LocalContainerLauncher.java:[line 524]
> M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
> dereference in
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.isJobNamePatternMatch(JobConf,
> String) due to return value of called method Dereferenced at
> MRAppMaster.java:[line 560]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6881) Fix warnings from Spotbugs in hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6881: Fix Version/s: 2.10.1 > Fix warnings from Spotbugs in hadoop-mapreduce > -- > > Key: MAPREDUCE-6881 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6881 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.0.0-alpha4, 2.10.1 > > Attachments: HADOOP-14339.001.patch, HADOOP-14339.002.patch, > HADOOP-14339.003.patch, HADOOP-14339.004.patch > > > Fix warnings from Spotbugs in hadoop-mapreduce since switched from findbugs > to spotbugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6881) Fix warnings from Spotbugs in hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175160#comment-17175160 ] Masatake Iwasaki commented on MAPREDUCE-6881: - cherry-picked to branch-2.10. > Fix warnings from Spotbugs in hadoop-mapreduce > -- > > Key: MAPREDUCE-6881 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6881 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Fix For: 3.0.0-alpha4, 2.10.1 > > Attachments: HADOOP-14339.001.patch, HADOOP-14339.002.patch, > HADOOP-14339.003.patch, HADOOP-14339.004.patch > > > Fix warnings from Spotbugs in hadoop-mapreduce since switched from findbugs > to spotbugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7286) Fix findbugs warnings in hadoop-mapreduce-project on branch-2.10
Masatake Iwasaki created MAPREDUCE-7286:
---
Summary: Fix findbugs warnings in hadoop-mapreduce-project on
branch-2.10
Key: MAPREDUCE-7286
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7286
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.10.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
{noformat}
$ find . -name findbugsXml.xml | xargs -n 1
/opt/findbugs-3.0.1/bin/convertXmlToText -longBugCodes
M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
dereference in org.apache.hadoop.examples.pi.Parser.parse(File, Map) due to
return value of called method Dereferenced at Parser.java:[line 70]
H P DM_BOXED_PRIMITIVE_FOR_COMPARE Bx: Primitive is boxed to call
Long.compareTo(Long): use Long.compare(long, long) instead At JVMId.java:[line
101]
M B ME_MUTABLE_ENUM_FIELD ME: org.apache.hadoop.mapred.Operation.jobACLNeeded
field is public and mutable In Operation.java
M B ME_MUTABLE_ENUM_FIELD ME: org.apache.hadoop.mapred.Operation.qACLNeeded
field is public and mutable In Operation.java
M D UC_USELESS_OBJECT UC: Useless object stored in variable paths of method
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager$HistoryFileInfo.moveToDone()
At HistoryFileManager.java:[line 416]
M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
dereference in new org.apache.hadoop.mapred.LocalContainerLauncher(AppContext,
TaskUmbilicalProtocol, ClassLoader) due to return value of called method
Dereferenced at LocalContainerLauncher.java:[line 124]
M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
dereference in
org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.relocalize() due
to return value of called method Dereferenced at
LocalContainerLauncher.java:[line 524]
M D NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE NP: Possible null pointer
dereference in
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.isJobNamePatternMatch(JobConf,
String) due to return value of called method Dereferenced at
MRAppMaster.java:[line 560]
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7285) Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7285:
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
> --
>
> Key: MAPREDUCE-7285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.4.0
>
>
> {noformat}
> [ebadger@foo bin]$ $HADOOP_HOME/bin/hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar
> sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1
> -r 1
> WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of
> HADOOP_PREFIX.
> java.lang.NoClassDefFoundError: junit/framework/TestCase
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:109)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:61)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {noformat}
> The sleep job continues to run after the error and succeeds successfully, but
> the error shouldn't be there. Something must have removed a jar or added an
> unfulfilled dependency on junit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7285) Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7285:
Status: Patch Available (was: Open)
> Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
> --
>
> Key: MAPREDUCE-7285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> [ebadger@foo bin]$ $HADOOP_HOME/bin/hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar
> sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1
> -r 1
> WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of
> HADOOP_PREFIX.
> java.lang.NoClassDefFoundError: junit/framework/TestCase
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:109)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:61)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {noformat}
> The sleep job continues to run after the error and succeeds successfully, but
> the error shouldn't be there. Something must have removed a jar or added an
> unfulfilled dependency on junit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7285) Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157820#comment-17157820
]
Masatake Iwasaki commented on MAPREDUCE-7285:
-
We need compile scope dependency on junit to bundle junit jar with dist, while
it should not be in usual runtime classpath. Only hadoop-dynamometer-infra has
this for running MiniDFSCluster now.
The error was caused on loading DistributedFSCheck extending junit TestCase.
How about just removing unnecessary junit dependency from the class? I created
[PR|https://github.com/apache/hadoop/pull/2139].
> Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
> --
>
> Key: MAPREDUCE-7285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> [ebadger@foo bin]$ $HADOOP_HOME/bin/hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar
> sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1
> -r 1
> WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of
> HADOOP_PREFIX.
> java.lang.NoClassDefFoundError: junit/framework/TestCase
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:109)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:61)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {noformat}
> The sleep job continues to run after the error and succeeds successfully, but
> the error shouldn't be there. Something must have removed a jar or added an
> unfulfilled dependency on junit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7285) Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
[
https://issues.apache.org/jira/browse/MAPREDUCE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned MAPREDUCE-7285:
---
Assignee: Masatake Iwasaki
> Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar
> --
>
> Key: MAPREDUCE-7285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> [ebadger@foo bin]$ $HADOOP_HOME/bin/hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar
> sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1
> -r 1
> WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of
> HADOOP_PREFIX.
> java.lang.NoClassDefFoundError: junit/framework/TestCase
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:109)
> at
> org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:61)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 21 more
> {noformat}
> The sleep job continues to run after the error and succeeds successfully, but
> the error shouldn't be there. Something must have removed a jar or added an
> unfulfilled dependency on junit
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7280) MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-7280: Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > MiniMRYarnCluster has hard-coded timeout waiting to start history server, > with no way to disable > > > Key: MAPREDUCE-7280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7280 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Nick Dimiduk >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.4.0 > > > Over in HBase, we've been chasing intermittent Jenkins failures in tests > involving MiniMRYarnCluster. In the latest incantion, HBASE-24493, we > finally tracked this down to a hard-coded 60sec timeout in MiniMRYarnCluster > on bringing up the JobHistoryServer... a feature we cannot disable for the > purpose of this test. We've had to disable running these tests for the time > being, which is less than ideal. > Would be great for MiniMRYarnCluster to (1) make JHS optional and/or (2) make > this timeout duration configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7281) Fix NoClassDefFoundError on 'mapred minicluster'
[
https://issues.apache.org/jira/browse/MAPREDUCE-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7281:
Fix Version/s: 3.3.1
> Fix NoClassDefFoundError on 'mapred minicluster'
>
>
> Key: MAPREDUCE-7281
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7281
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
>
> {noformat}
> $ bin/mapred minicluster
> 2020-06-17 12:01:29,133 INFO mapreduce.MiniHadoopClusterManager: Updated 0
> configuration settings from command line.
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:298)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:242)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:251)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.(MiniDFSCluster.java:224)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7281) Fix NoClassDefFoundError on 'mapred minicluster'
[
https://issues.apache.org/jira/browse/MAPREDUCE-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7281:
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix NoClassDefFoundError on 'mapred minicluster'
>
>
> Key: MAPREDUCE-7281
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7281
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.4.0
>
>
> {noformat}
> $ bin/mapred minicluster
> 2020-06-17 12:01:29,133 INFO mapreduce.MiniHadoopClusterManager: Updated 0
> configuration settings from command line.
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:298)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:242)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:251)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.(MiniDFSCluster.java:224)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7281) Fix NoClassDefFoundError on 'mapred minicluster'
[
https://issues.apache.org/jira/browse/MAPREDUCE-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7281:
Status: Patch Available (was: Open)
> Fix NoClassDefFoundError on 'mapred minicluster'
>
>
> Key: MAPREDUCE-7281
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7281
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> $ bin/mapred minicluster
> 2020-06-17 12:01:29,133 INFO mapreduce.MiniHadoopClusterManager: Updated 0
> configuration settings from command line.
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:298)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:242)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:251)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.(MiniDFSCluster.java:224)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7281) Fix NoClassDefFoundError on 'mapred minicluster'
[
https://issues.apache.org/jira/browse/MAPREDUCE-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki moved HADOOP-17073 to MAPREDUCE-7281:
--
Key: MAPREDUCE-7281 (was: HADOOP-17073)
Project: Hadoop Map/Reduce (was: Hadoop Common)
> Fix NoClassDefFoundError on 'mapred minicluster'
>
>
> Key: MAPREDUCE-7281
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7281
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> $ bin/mapred minicluster
> 2020-06-17 12:01:29,133 INFO mapreduce.MiniHadoopClusterManager: Updated 0
> configuration settings from command line.
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:298)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:242)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:251)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.(MiniDFSCluster.java:224)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7280) MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-7280: Status: Patch Available (was: Open) > MiniMRYarnCluster has hard-coded timeout waiting to start history server, > with no way to disable > > > Key: MAPREDUCE-7280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7280 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Nick Dimiduk >Assignee: Masatake Iwasaki >Priority: Major > > Over in HBase, we've been chasing intermittent Jenkins failures in tests > involving MiniMRYarnCluster. In the latest incantion, HBASE-24493, we > finally tracked this down to a hard-coded 60sec timeout in MiniMRYarnCluster > on bringing up the JobHistoryServer... a feature we cannot disable for the > purpose of this test. We've had to disable running these tests for the time > being, which is less than ideal. > Would be great for MiniMRYarnCluster to (1) make JHS optional and/or (2) make > this timeout duration configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7280) MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable
[ https://issues.apache.org/jira/browse/MAPREDUCE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki moved HADOOP-17071 to MAPREDUCE-7280: -- Component/s: (was: test) test Key: MAPREDUCE-7280 (was: HADOOP-17071) Project: Hadoop Map/Reduce (was: Hadoop Common) > MiniMRYarnCluster has hard-coded timeout waiting to start history server, > with no way to disable > > > Key: MAPREDUCE-7280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7280 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Nick Dimiduk >Assignee: Masatake Iwasaki >Priority: Major > > Over in HBase, we've been chasing intermittent Jenkins failures in tests > involving MiniMRYarnCluster. In the latest incantion, HBASE-24493, we > finally tracked this down to a hard-coded 60sec timeout in MiniMRYarnCluster > on bringing up the JobHistoryServer... a feature we cannot disable for the > purpose of this test. We've had to disable running these tests for the time > being, which is less than ideal. > Would be great for MiniMRYarnCluster to (1) make JHS optional and/or (2) make > this timeout duration configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7256) Fix javadoc error in SimpleExponentialSmoothing
[
https://issues.apache.org/jira/browse/MAPREDUCE-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015605#comment-17015605
]
Masatake Iwasaki commented on MAPREDUCE-7256:
-
Thanks, [~aajisaka].
> Fix javadoc error in SimpleExponentialSmoothing
> ---
>
> Key: MAPREDUCE-7256
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7256
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
>
> {{-Pdist}} build fails due to javadoc error on
> SimpleExponentialSmoothing.java.
> {noformat}
> [ERROR]
> /ext/srcs/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
> error: bad use of '>'
> [ERROR]* @return true if we have number of samples > kMinimumReads and
> the record
> [ERROR]^
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7257) Fix incorrect javadoc format
[
https://issues.apache.org/jira/browse/MAPREDUCE-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015572#comment-17015572
]
Masatake Iwasaki commented on MAPREDUCE-7257:
-
[~aajisaka] I filed the same issue on MAPREDUCE-7256.
> Fix incorrect javadoc format
>
>
> Key: MAPREDUCE-7257
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7257
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: documentation
>Reporter: Akira Ajisaka
>Priority: Major
> Labels: newbie
>
> `mvn package -Pdist -DskipTests` fails.
> {noformat}
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
> error: bad use of '>'
> [ERROR]* @return true if we have number of samples > kMinimumReads and
> the record
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7256) Fix javadoc error in SimpleExponentialSmoothing
[
https://issues.apache.org/jira/browse/MAPREDUCE-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-7256:
Target Version/s: 3.3.0
Status: Patch Available (was: Open)
> Fix javadoc error in SimpleExponentialSmoothing
> ---
>
> Key: MAPREDUCE-7256
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7256
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> {{-Pdist}} build fails due to javadoc error on
> SimpleExponentialSmoothing.java.
> {noformat}
> [ERROR]
> /ext/srcs/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
> error: bad use of '>'
> [ERROR]* @return true if we have number of samples > kMinimumReads and
> the record
> [ERROR]^
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7256) Fix javadoc error in SimpleExponentialSmoothing
Masatake Iwasaki created MAPREDUCE-7256:
---
Summary: Fix javadoc error in SimpleExponentialSmoothing
Key: MAPREDUCE-7256
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7256
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
{{-Pdist}} build fails due to javadoc error on SimpleExponentialSmoothing.java.
{noformat}
[ERROR]
/ext/srcs/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
error: bad use of '>'
[ERROR]* @return true if we have number of samples > kMinimumReads and the
record
[ERROR]^
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6521:
Fix Version/s: 2.10.0
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 2.10.0, 3.3.0
>
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888421#comment-16888421
]
Masatake Iwasaki commented on MAPREDUCE-6521:
-
Thanks [~ste...@apache.org]. We have same issue in branch-2 and the patch is
applicable as is. I tested it by running {{mvn test
-Dtest=TestMiniMRChildTask}} before and after cherry-picking the fix. I'm going
to push this to branch-2.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6794) Remove unused properties from TTConfig.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858199#comment-16858199 ] Masatake Iwasaki commented on MAPREDUCE-6794: - +1. Thanks for the update [~aajisaka]. > Remove unused properties from TTConfig.java > --- > > Key: MAPREDUCE-6794 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6794 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: MAPREDUCE-6794.01.patch, MAPREDUCE-6794.02.patch, > MAPREDUCE-6794.03.patch, MAPREDUCE-6794.04.patch > > > There are many unused properties in TTConfig.java. Let's remove them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6794) Remove unused properties from TTConfig.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856525#comment-16856525
]
Masatake Iwasaki commented on MAPREDUCE-6794:
-
{noformat}
-| slave.host.name | dfs.datanode.hostname |
{noformat}
[~aajisaka] This one is not related to TaskTracker. {{dfs.datanode.hostname}}
is still used.
I'm +1 if this is addressed.
TestAMSimulator is flaky one. It passed on my local with 03 applied.
TTConfig is marked as InterfaceAudience.Private and I could not find its usage
in major downstrems.
> Remove unused properties from TTConfig.java
> ---
>
> Key: MAPREDUCE-6794
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6794
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: MAPREDUCE-6794.01.patch, MAPREDUCE-6794.02.patch,
> MAPREDUCE-6794.03.patch
>
>
> There are many unused properties in TTConfig.java. Let's remove them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7198) mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
[
https://issues.apache.org/jira/browse/MAPREDUCE-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836748#comment-16836748
]
Masatake Iwasaki commented on MAPREDUCE-7198:
-
Thanks for the update. +1.
> mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
> ---
>
> Key: MAPREDUCE-7198
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7198
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: MAPREDUCE-7198.01.patch, MAPREDUCE-7198.02.patch
>
>
> mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
> after MAPREDUCE-6190. If the task timeout is configured as zero the task
> fails with stuck timeout, if the TaskStatus is null.
> {code}
> if (sendProgress) {
> // we need to send progress update
> updateCounters();
> checkTaskLimits();
> taskStatus.statusUpdate(taskProgress.get(),
> taskProgress.toString(),
> counters);
> amFeedback = umbilical.statusUpdate(taskId, taskStatus);
> taskFound = amFeedback.getTaskFound();
> taskStatus.clearStatus();
> }
> else {
> // send ping
> amFeedback = umbilical.statusUpdate(taskId, null);
> taskFound = amFeedback.getTaskFound();
> }
> {code}
> This issue is reported by [~bibinchundatt].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7198) mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
[
https://issues.apache.org/jira/browse/MAPREDUCE-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829958#comment-16829958
]
Masatake Iwasaki commented on MAPREDUCE-7198:
-
Should "mapreduce.task.stuck.timeout-ms = 0" mean disabling the timeout rather
than "mapreduce.task.timeout = 0"? If the goal here is to retain previous
behaviour before MAPREDUCE-6190 as task does not timeout only if
mapreduce.task.timeout = 0, the task-stuck-timeout should be disabled by
default (by set the value to 0).
> mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
> ---
>
> Key: MAPREDUCE-7198
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7198
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: MAPREDUCE-7198.01.patch
>
>
> mapreduce.task.timeout=0 configuration used to disable timeout doesn't work
> after MAPREDUCE-6190. If the task timeout is configured as zero the task
> fails with stuck timeout, if the TaskStatus is null.
> {code}
> if (sendProgress) {
> // we need to send progress update
> updateCounters();
> checkTaskLimits();
> taskStatus.statusUpdate(taskProgress.get(),
> taskProgress.toString(),
> counters);
> amFeedback = umbilical.statusUpdate(taskId, taskStatus);
> taskFound = amFeedback.getTaskFound();
> taskStatus.clearStatus();
> }
> else {
> // send ping
> amFeedback = umbilical.statusUpdate(taskId, null);
> taskFound = amFeedback.getTaskFound();
> }
> {code}
> This issue is reported by [~bibinchundatt].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6644) Use doxia macro to generate in-page TOC of MapReduce site documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6644: Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.9.0 Status: Resolved (was: Patch Available) Committed this to branch-2 and trunk. Thanks for the review, [~ajisakaa]. > Use doxia macro to generate in-page TOC of MapReduce site documentation > --- > > Key: MAPREDUCE-6644 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6644 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: MAPREDUCE-6644.001.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6644) Use doxia macro to generate in-page TOC of MapReduce site documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6644: Status: Patch Available (was: Open) > Use doxia macro to generate in-page TOC of MapReduce site documentation > --- > > Key: MAPREDUCE-6644 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6644 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Attachments: MAPREDUCE-6644.001.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6644) Use doxia macro to generate in-page TOC of MapReduce site documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6644: Attachment: MAPREDUCE-6644.001.patch Since HADOOP-12470 came in, we can use toc macro in MR docs. > Use doxia macro to generate in-page TOC of MapReduce site documentation > --- > > Key: MAPREDUCE-6644 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6644 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > Attachments: MAPREDUCE-6644.001.patch > > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6644) Use doxia macro to generate in-page TOC of MapReduce site documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6644: Description: Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. (was: Since maven-site-plugin 3.5 was releaced, we can use toc macro in Markdown.) > Use doxia macro to generate in-page TOC of MapReduce site documentation > --- > > Key: MAPREDUCE-6644 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6644 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.7.0 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki > > Since maven-site-plugin 3.5 was released, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6644) Use doxia macro to generate in-page TOC of MapReduce site documentation
Masatake Iwasaki created MAPREDUCE-6644: --- Summary: Use doxia macro to generate in-page TOC of MapReduce site documentation Key: MAPREDUCE-6644 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6644 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.7.0 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Since maven-site-plugin 3.5 was releaced, we can use toc macro in Markdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979525#comment-14979525
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
bq. .. we should use Configuration to enable timeline server instead of using
enableAHS
I agree but the constructer has already marked as deprecated by YARN-3573. I
think there is nothing we can do more. We should not remove public constructor
of mini cluster because it is possible to be used by downstream projects.
This jira is addressing the issue about not AHS but JHS in MiniMRYarnCluster:-)
Though {{JHAdminConfig}} does not have on/off switch like
{{YarnConfiguration.TIMELINE_SERVICE_ENABLED}}, I don't think we need to add it.
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Attachments: MAPREDUCE-6525.001.patch
>
>
> MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with
> configuration of existing MiniMRYarnCluster but the address of HistoryServer
> is properly set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Attachment: MAPREDUCE-6525.001.patch I attached the 001. The copy and logging of configuration should be done after all services are started. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Description: MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with configuration of existing MiniMRYarnCluster but the address of HistoryServer is properly set. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6525: Status: Patch Available (was: Open) > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976029#comment-14976029
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
{{MiniMRYarnClusterAdapter#restart}} creates new MiniMRYarnCluster with
configuration of existing MiniMRYarnCluster. MiniMRYarnCluster tries to copy
the value of {{JHAdminConfig.MR_HISTORY_ADDRESS}} in the configuration of
JobHistoryServer (updated by using {{Configuration#updateConnectAddr}}) in
startup.
{code}
//need to do this because historyServer.init creates a new Configuration
getConfig().set(JHAdminConfig.MR_HISTORY_ADDRESS,
historyServer.getConfig().get(JHAdminConfig.MR_HISTORY_ADDRESS));
MRWebAppUtil.setJHSWebappURLWithoutScheme(getConfig(),
MRWebAppUtil.getJHSWebappURLWithoutScheme(historyServer.getConfig()));
{code}
According to the log of minicluster, the updated value seems not yet to be
visible at above point.
before restart:
{noformat}
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(248)) - MiniMRYARN ResourceManager
address: centos7:0
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(250)) - MiniMRYARN ResourceManager web
address: centos7:0
2015-10-27 17:27:52,825 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(252)) - MiniMRYARN HistoryServer address:
centos7:0
2015-10-27 17:27:52,826 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(254)) - MiniMRYARN HistoryServer web
address: localhost:0
{noformat}
after restart:
{noformat}
2015-10-27 17:27:55,330 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(198)) - MiniMRYARN ResourceManager
address: localhost:45359
2015-10-27 17:27:55,330 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(200)) - MiniMRYARN ResourceManager web
address: centos7:34289
2015-10-27 17:27:55,331 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(202)) - MiniMRYARN HistoryServer address:
localhost:52822
2015-10-27 17:27:55,331 INFO [main] v2.MiniMRYarnCluster
(MiniMRYarnCluster.java:serviceStart(204)) - MiniMRYARN HistoryServer web
address: localhost:57028
{noformat}
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved MAPREDUCE-6524. - Resolution: Duplicate The cause seems to be same with MAPREDUCE-6525 and the patch of it fixed this too. I'm closing this as duplicate. > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > > The historyClient tried to connect to port 0 and failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976091#comment-14976091
]
Masatake Iwasaki commented on MAPREDUCE-6521:
-
The failure of TestMRJobsWithHistoryService and TestMiniMRClientCluster is not
related to the patch. I filed MAPREDUCE-6524 and MAPREDUCE-6525 for them.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6524: Description: The historyClient tried to connect to port 0 and failed. > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > > The historyClient tried to connect to port 0 and failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[
https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976083#comment-14976083
]
Masatake Iwasaki commented on MAPREDUCE-6524:
-
{noformat}
Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 95.533 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
testJobHistoryData(org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService)
Time elapsed: 94.426 sec <<< ERROR!
java.io.IOException: java.net.ConnectException: Call From centos7/127.0.0.1 to
centos7:0 failed on connection exception: java.net.ConnectException: Connection
refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
...
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.testJobHistoryData(TestMRJobsWithHistoryService.java:153)
{noformat}
> Fix intermittent test failure of
> TestMRJobsWithHistoryService.testJobHistoryData
>
>
> Key: MAPREDUCE-6524
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> The historyClient tried to connect to port 0 and failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976679#comment-14976679 ] Masatake Iwasaki commented on MAPREDUCE-6525: - s/similar to MAPREDUCE-6525/similar to MAPREDUCE-6524/ > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[ https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976676#comment-14976676 ] Masatake Iwasaki commented on MAPREDUCE-6525: - The failure of TestJobName is not related because it does not use mini cluster. The error message of TestNonExistentJob is similar to MAPREDUCE-6525 and it implies that the fix is not enough. > Fix test failure of TestMiniMRClientCluster.testRestart > --- > > Key: MAPREDUCE-6525 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: MAPREDUCE-6525.001.patch > > > MiniMRYarnClusterAdapter#restart creates new MiniMRYarnCluster with > configuration of existing MiniMRYarnCluster but the address of HistoryServer > is properly set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6524: Component/s: test > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
Masatake Iwasaki created MAPREDUCE-6525: --- Summary: Fix test failure of TestMiniMRClientCluster.testRestart Key: MAPREDUCE-6525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
Masatake Iwasaki created MAPREDUCE-6524: --- Summary: Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData Key: MAPREDUCE-6524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975543#comment-14975543
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
{noformat}
Running org.apache.hadoop.mapred.TestMiniMRClientCluster
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 71.73 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestMiniMRClientCluster
testRestart(org.apache.hadoop.mapred.TestMiniMRClientCluster) Time elapsed:
28.472 sec <<< FAILURE!
org.junit.ComparisonFailure: Address before restart: centos7:0 is different
from new address: localhost:37880 expected:<[centos7:]0> but
was:<[localhost:3788]0>
at org.junit.Assert.assertEquals(Assert.java:115)
at
org.apache.hadoop.mapred.TestMiniMRClientCluster.testRestart(TestMiniMRClientCluster.java:146)
{noformat}
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6521:
Status: Patch Available (was: Open)
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6521:
Attachment: MAPREDUCE-6521.001.patch
I attached the patch. I verified that /tmp/hadoop-yarn is not created by
manually running some unit tests which uses MiniMRYarnCluster with local
filesystem.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968794#comment-14968794
]
Masatake Iwasaki commented on MAPREDUCE-6521:
-
{code}
if (conf.get(MRJobConfig.MR_AM_STAGING_DIR) == null) {
conf.set(MRJobConfig.MR_AM_STAGING_DIR, new File(getTestWorkDir(),
"apps_staging_dir/").getAbsolutePath());
}
{code}
The intent of the if statement seems to be "if
yarn.app.mapreduce.am.staging-dir is not specified in configuration" but the
{{conf.get(MRJobConfig.MR_AM_STAGING_DIR)}} always returns non-null value
because the default value is defined in mapred-default.xml as
{{/tmp/hadoop-yarn/staging}}.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
Masatake Iwasaki created MAPREDUCE-6521:
---
Summary: MiniMRYarnCluster should not create
/tmp/hadoop-yarn/staging on local filesystem in unit test
Key: MAPREDUCE-6521
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
should be under {{testWorkDir}} if the file system is localFs in order to make
it to be removed by {{mvn clean}}. It would also avoid issues under parallel
unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-1125) SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
[
https://issues.apache.org/jira/browse/MAPREDUCE-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-1125:
Attachment: MAPREDUCE-1125.004.patch
I attached rebased patch. I left the version of old signature for backward
compatibility. There is no test code for .cc in the current tree of hadoop-pipes
SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh
---
Key: MAPREDUCE-1125
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1125
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 0.21.0
Reporter: Simone Leo
Assignee: Simone Leo
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-1125-2.patch, MAPREDUCE-1125-3.patch,
MAPREDUCE-1125.004.patch
{noformat}
*** SerialUtils.hh ***
float deserializeFloat(InStream stream);
*** SerialUtils.cc ***
void deserializeFloat(float t, InStream stream)
{
char buf[sizeof(float)];
stream.read(buf, sizeof(float));
XDR xdrs;
xdrmem_create(xdrs, buf, sizeof(float), XDR_DECODE);
xdr_float(xdrs, t);
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4065: Resolution: Won't Fix Status: Resolved (was: Patch Available) I'm closing this as won't fix because I think the release source tarball containing *.proto is enough. Feel free to reopen this, [~rcastain] and [~ozawa]. Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi Ozawa Attachments: MAPREDUCE-4065.002.patch, MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4065: Labels: (was: BB2015-05-RFC) Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi Ozawa Attachments: MAPREDUCE-4065.002.patch, MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5690) TestLocalMRNotification.testMR occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-5690:
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Closing this as duplicate of MAPREDUCE-4376
TestLocalMRNotification.testMR occasionally fails
-
Key: MAPREDUCE-5690
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5690
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Attachments: MAPREDUCE-5690.1.diff
TestLocalMRNotificationis occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.mapred.TestLocalMRNotification
---
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.992 sec
FAILURE! - in org.apache.hadoop.mapred.TestLocalMRNotification
testMR(org.apache.hadoop.mapred.TestLocalMRNotification) Time elapsed:
24.881 sec ERROR!
java.io.IOException: Job cleanup didn't start in 20 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobKill(UtilsForTests.java:685)
at
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4978: Resolution: Won't Fix Status: Resolved (was: Patch Available) This patch adds code to set properties listed below. * map.input.file * map.input.start * map.input.length Few part of current MapReduce code use these. * Those are used by CombineFileRecordReader but it set them by itself * map.input.file is used by MultipleOutputFormat which is old api I am closing this as Won't fix. Please reopen this if you need this, [~liangly]. Add a updateJobWithSplit() method for new-api job - Key: MAPREDUCE-4978 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 1.1.2 Reporter: Liyin Liang Assignee: Liyin Liang Attachments: 4978-1.diff HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api job. It's better to add another method for new-api job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5690) TestLocalMRNotification.testMR occasionally fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-5690:
Labels: (was: BB2015-05-TBR)
TestLocalMRNotification.testMR occasionally fails
-
Key: MAPREDUCE-5690
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5690
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Liyin Liang
Assignee: Liyin Liang
Attachments: MAPREDUCE-5690.1.diff
TestLocalMRNotificationis occasionally failing with the error:
{code}
---
Test set: org.apache.hadoop.mapred.TestLocalMRNotification
---
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.992 sec
FAILURE! - in org.apache.hadoop.mapred.TestLocalMRNotification
testMR(org.apache.hadoop.mapred.TestLocalMRNotification) Time elapsed:
24.881 sec ERROR!
java.io.IOException: Job cleanup didn't start in 20 seconds
at
org.apache.hadoop.mapred.UtilsForTests.runJobKill(UtilsForTests.java:685)
at
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4978: Labels: (was: BB2015-05-TBR) Add a updateJobWithSplit() method for new-api job - Key: MAPREDUCE-4978 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 1.1.2 Reporter: Liyin Liang Assignee: Liyin Liang Attachments: 4978-1.diff HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api job. It's better to add another method for new-api job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5626) TaskLogServlet could not get syslog
[
https://issues.apache.org/jira/browse/MAPREDUCE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-5626:
Labels: patch (was: BB2015-05-TBR patch)
TaskLogServlet could not get syslog
---
Key: MAPREDUCE-5626
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5626
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.1
Environment: Linux version 2.6.18-238.9.1.el5
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
hadoop-1.2.1
Reporter: yangjun
Priority: Minor
Labels: patch
Fix For: 1.2.1
Original Estimate: 2h
Remaining Estimate: 2h
When multiply tasks use one jvm and generated logs.
eg.
./attempt_201211220735_0001_m_00_0:
log.index
./attempt_201211220735_0001_m_01_0:
log.index
./attempt_201211220735_0001_m_02_0:
log.index stderr stdout syslog
get from http://:50060/tasklog?attemptid=
attempt_201211220735_0001_m_00_0
could get stderr,stdout,but not the others,include syslog.
see TaskLogServlet.haveTaskLog() method, not check from local log.index,
but check the original path.
resolve:
modify TaskLogServlet haveTaskLog method
private boolean haveTaskLog(TaskAttemptID taskId, boolean isCleanup,
TaskLog.LogName type) throws IOException {
File f = TaskLog.getTaskLogFile(taskId, isCleanup, type);
if (f.exists() f.canRead()) {
return true;
} else {
File indexFile = TaskLog.getIndexFile(taskId, isCleanup);
if (!indexFile.exists()) {
return false;
}
BufferedReader fis;
try {
fis = new BufferedReader(new InputStreamReader(
SecureIOUtils.openForRead(indexFile,
TaskLog.obtainLogDirOwner(taskId;
} catch (FileNotFoundException ex) {
LOG.warn(Index file for the log of + taskId
+ does not exist.);
// Assume no task reuse is used and files exist on attemptdir
StringBuffer input = new StringBuffer();
input.append(LogFileDetail.LOCATION
+ TaskLog.getAttemptDir(taskId, isCleanup) + \n);
for (LogName logName : TaskLog.LOGS_TRACKED_BY_INDEX_FILES) {
input.append(logName + :0 -1\n);
}
fis = new BufferedReader(new StringReader(input.toString()));
}
try {
String str = fis.readLine();
if (str == null) { // thefile doesn't have anything
throw new IOException(Index file for the log of +
taskId
+ is empty.);
}
String loc =
str.substring(str.indexOf(LogFileDetail.LOCATION)
+ LogFileDetail.LOCATION.length());
File tf = new File(loc, type.toString());
return tf.exists() tf.canRead();
} finally {
if (fis != null)
fis.close();
}
}
}
workaround:
url add filter=SYSLOG could print syslog also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5626) TaskLogServlet could not get syslog
[
https://issues.apache.org/jira/browse/MAPREDUCE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-5626:
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
I think this could be closed as won't fix.
[~yangj...@sohu.com], Could you attach patch file to JIRA as described in
[wiki|https://wiki.apache.org/hadoop/HowToContribute], if you have update for
this or another issue?
TaskLogServlet could not get syslog
---
Key: MAPREDUCE-5626
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5626
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.1
Environment: Linux version 2.6.18-238.9.1.el5
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
hadoop-1.2.1
Reporter: yangjun
Priority: Minor
Labels: patch
Fix For: 1.2.1
Original Estimate: 2h
Remaining Estimate: 2h
When multiply tasks use one jvm and generated logs.
eg.
./attempt_201211220735_0001_m_00_0:
log.index
./attempt_201211220735_0001_m_01_0:
log.index
./attempt_201211220735_0001_m_02_0:
log.index stderr stdout syslog
get from http://:50060/tasklog?attemptid=
attempt_201211220735_0001_m_00_0
could get stderr,stdout,but not the others,include syslog.
see TaskLogServlet.haveTaskLog() method, not check from local log.index,
but check the original path.
resolve:
modify TaskLogServlet haveTaskLog method
private boolean haveTaskLog(TaskAttemptID taskId, boolean isCleanup,
TaskLog.LogName type) throws IOException {
File f = TaskLog.getTaskLogFile(taskId, isCleanup, type);
if (f.exists() f.canRead()) {
return true;
} else {
File indexFile = TaskLog.getIndexFile(taskId, isCleanup);
if (!indexFile.exists()) {
return false;
}
BufferedReader fis;
try {
fis = new BufferedReader(new InputStreamReader(
SecureIOUtils.openForRead(indexFile,
TaskLog.obtainLogDirOwner(taskId;
} catch (FileNotFoundException ex) {
LOG.warn(Index file for the log of + taskId
+ does not exist.);
// Assume no task reuse is used and files exist on attemptdir
StringBuffer input = new StringBuffer();
input.append(LogFileDetail.LOCATION
+ TaskLog.getAttemptDir(taskId, isCleanup) + \n);
for (LogName logName : TaskLog.LOGS_TRACKED_BY_INDEX_FILES) {
input.append(logName + :0 -1\n);
}
fis = new BufferedReader(new StringReader(input.toString()));
}
try {
String str = fis.readLine();
if (str == null) { // thefile doesn't have anything
throw new IOException(Index file for the log of +
taskId
+ is empty.);
}
String loc =
str.substring(str.indexOf(LogFileDetail.LOCATION)
+ LogFileDetail.LOCATION.length());
File tf = new File(loc, type.toString());
return tf.exists() tf.canRead();
} finally {
if (fis != null)
fis.close();
}
}
}
workaround:
url add filter=SYSLOG could print syslog also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-695) MiniMRCluster while shutting down should not wait for currently running jobs to finish
[
https://issues.apache.org/jira/browse/MAPREDUCE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-695:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
MiniMRCluster is wrapper of MiniMRYarnCluster now and {{waitTaskTrackers}} is
not used from anywhere.
{code}
private void waitTaskTrackers() {
throw new UnsupportedOperationException();
}
{code}
I'm closing this as won't fix.
MiniMRCluster while shutting down should not wait for currently running jobs
to finish
--
Key: MAPREDUCE-695
URL: https://issues.apache.org/jira/browse/MAPREDUCE-695
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.0.3
Reporter: Sreekanth Ramakrishnan
Priority: Minor
Attachments: mapreduce-695.patch
Currently in {{org.apache.hadoop.mapred.MiniMRCluster.shutdown()}} we do a
{{waitTaskTrackers()}} which can cause {{MiniMRCluster}} to hang indefinitely
when used in conjunction with Controlled jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-695) MiniMRCluster while shutting down should not wait for currently running jobs to finish
[
https://issues.apache.org/jira/browse/MAPREDUCE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-695:
---
Labels: (was: BB2015-05-TBR)
MiniMRCluster while shutting down should not wait for currently running jobs
to finish
--
Key: MAPREDUCE-695
URL: https://issues.apache.org/jira/browse/MAPREDUCE-695
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 1.0.3
Reporter: Sreekanth Ramakrishnan
Priority: Minor
Attachments: mapreduce-695.patch
Currently in {{org.apache.hadoop.mapred.MiniMRCluster.shutdown()}} we do a
{{waitTaskTrackers()}} which can cause {{MiniMRCluster}} to hang indefinitely
when used in conjunction with Controlled jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4065: Labels: (was: BB2015-05-TBR) Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi Ozawa Attachments: MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4065: Labels: BB2015-05-RFC (was: ) Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi Ozawa Labels: BB2015-05-RFC Attachments: MAPREDUCE-4065.002.patch, MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-4065: Attachment: MAPREDUCE-4065.002.patch I updated the patch, putting files under same maven sub-project to same sub-directory, excluding files under */src/test/proto/. Though I updated this, I think that the source tarball containing *.proto is enough now... Anyway, I attached the patch. I ask for committer's decision to just close this as won't fix or not. Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2, 2.4.0 Reporter: Ralph H Castain Assignee: Tsuyoshi Ozawa Attachments: MAPREDUCE-4065.002.patch, MAPREDUCE-4065.1.patch Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343969#comment-14343969 ] Masatake Iwasaki commented on MAPREDUCE-6260: - I submitted the backport patch in HADOOP-11633. Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Fix For: 3.0.0 Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335758#comment-14335758 ] Masatake Iwasaki commented on MAPREDUCE-6260: - I am going to make branch-2 patch for this. Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Fix For: 3.0.0 Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14324773#comment-14324773 ] Masatake Iwasaki commented on MAPREDUCE-6260: - Thanks [~aw]! Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Fix For: 3.0.0 Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6260: Status: Patch Available (was: Open) Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323213#comment-14323213 ] Masatake Iwasaki commented on MAPREDUCE-6260: - You can see build doc at http://iwasakims.github.io/MAPREDUCE-6260/hadoop-project/ Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6260: Attachment: MAPREDUCE-6260-01.patch I attached updated patch including fixes as: * fixed TOC of each page. * *.vm: added workaround for ### is removed as comment by velocity template * EncryptedShuffle.md: fixed broken link to subsection * MapredAppMasterRest.md: fixed URL sample not in code block * MapReduceTutorial.md: fixed fomatting of broken code block * MapReduceTutorial.md: removed garbage white spaces Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6260: Component/s: documentation Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Attachments: MAPREDUCE-6260-00.patch, MAPREDUCE-6260-01.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321547#comment-14321547 ] Masatake Iwasaki commented on MAPREDUCE-6260: - I am going to take over updating 00. Please ping me if anyone already have progress on this. Thanks. Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: MAPREDUCE-6260-00.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6260) Convert site documentation to markdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned MAPREDUCE-6260: --- Assignee: Masatake Iwasaki Convert site documentation to markdown -- Key: MAPREDUCE-6260 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6260 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Masatake Iwasaki Attachments: MAPREDUCE-6260-00.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) TestHighRamJob fails due to the change in MAPREDUCE-5785
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Summary: TestHighRamJob fails due to the change in MAPREDUCE-5785 (was:
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml)
TestHighRamJob fails due to the change in MAPREDUCE-5785
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch, MAPREDUCE-6234.002.patch,
MAPREDUCE-6234.003.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Attachment: MAPREDUCE-6234.003.patch
003 fixes test failure without changing the value of DEFAULT_*_MEMORY_MB.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch, MAPREDUCE-6234.002.patch,
MAPREDUCE-6234.003.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310778#comment-14310778
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
Thanks for the clarification, [~varun_saxena]. I might misunderstand that you
don't agree the change by comment beggining with -1:)
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch, MAPREDUCE-6234.002.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309665#comment-14309665
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
[~varun_saxena], some tests need not effective default value but value
literally set in conf. The name DEFAULT_MAP_MEMORY_MB implies the default value
of mapreduce.map.memory.mb. 1024 is the value defermined from
mapreduce.map.memory.mb *and* mapreduce.map.java.opts. If we need the 1024
in some constants, it should not be named as DEFAULT_MAP_MEMORY_MB. The .002
patch addresses this.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch, MAPREDUCE-6234.002.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6223) TestJobConf#testNegativeValueForTaskVmem failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308760#comment-14308760
]
Masatake Iwasaki commented on MAPREDUCE-6223:
-
s/not local value but//
TestJobConf#testNegativeValueForTaskVmem failures
-
Key: MAPREDUCE-6223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6223
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Gera Shegalov
Assignee: Varun Saxena
Attachments: MAPREDUCE-6223.001.patch, MAPREDUCE-6223.002.patch
{code}
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.328 sec
FAILURE! - in org.apache.hadoop.conf.TestJobConf
testNegativeValueForTaskVmem(org.apache.hadoop.conf.TestJobConf) Time
elapsed: 0.089 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Attachment: MAPREDUCE-6234.002.patch
.002 addresses my comment in MAPREDUCE-6223. Tests needing default value in
conf can use {{MRJobConfig.DEFAULT_MAP_MEMORY_MB}} and test needing the value
processed by JobConf#getMemoryRequired can use
{{JobConf.DEFAULT_MAP_MEMORY_REQUIRED}}.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch, MAPREDUCE-6234.002.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6223) TestJobConf#testNegativeValueForTaskVmem failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308748#comment-14308748
]
Masatake Iwasaki commented on MAPREDUCE-6223:
-
I think JobConf#getMemoryReuiqred should get 1024 from not local value but
constant in MRJobConfig other than DEFAULT_*_MEMORY_MB because 1024 is never
set in Configuration. [~ajisakaa] / [~ozawa], please commit the patch of this
issue first. I will update the patch of MAPREDUCE-6234 later.
TestJobConf#testNegativeValueForTaskVmem failures
-
Key: MAPREDUCE-6223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6223
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Gera Shegalov
Assignee: Varun Saxena
Attachments: MAPREDUCE-6223.001.patch, MAPREDUCE-6223.002.patch
{code}
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.328 sec
FAILURE! - in org.apache.hadoop.conf.TestJobConf
testNegativeValueForTaskVmem(org.apache.hadoop.conf.TestJobConf) Time
elapsed: 0.089 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308487#comment-14308487
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
I agree that the code of gridmix should be updated. But I think it is very
confusing that MRJobConfig.DEFAULT_MAP_MEMORY_MB is not same with the value of
mapreduce.map.memory.mb in mapred-default.xml and it should be fixed without
regarding to the test failure of gridmix.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6243: Attachment: MAPREDUCE-6243.002.patch updated the patch. Thanks. Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Masatake Iwasaki Priority: Minor Labels: newbie Attachments: MAPREDUCE-6243.001.patch, MAPREDUCE-6243.002.patch, findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6243: Status: Patch Available (was: Open) Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Masatake Iwasaki Priority: Minor Labels: newbie Attachments: MAPREDUCE-6243.001.patch, findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6243: Attachment: MAPREDUCE-6243.001.patch attached patch. Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Masatake Iwasaki Priority: Minor Labels: newbie Attachments: MAPREDUCE-6243.001.patch, findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned MAPREDUCE-6243: --- Assignee: Masatake Iwasaki Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Masatake Iwasaki Priority: Minor Labels: newbie Attachments: findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Component/s: mrv2
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Status: Patch Available (was: Open)
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6236) Fix findbugs warnings in hadoop-mapreduce-client-core
[
https://issues.apache.org/jira/browse/MAPREDUCE-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved MAPREDUCE-6236.
-
Resolution: Duplicate
Yes. I closed this as duplicate. Thanks [~varun_saxena].
Fix findbugs warnings in hadoop-mapreduce-client-core
-
Key: MAPREDUCE-6236
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6236
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
{code}
DmFound reliance on default encoding in
org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[]):
java.io.ByteArrayOutputStream.toString()
DmFound reliance on default encoding in
org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[]): new
java.io.PrintStream(OutputStream)
BxBoxing/unboxing to parse a primitive
org.apache.hadoop.mapred.TaskLogAppender.setOptionsFromSystemProperties()
Bx
org.apache.hadoop.mapreduce.lib.fieldsel.FieldSelectionHelper.extractFields(String[],
List) invokes inefficient new Integer(String) constructor; use
Integer.valueOf(String) instead
RCN Redundant nullcheck of info, which is known to be non-null in
org.apache.hadoop.mapred.IndexCache.removeMap(String)
RCN Redundant nullcheck of
org.apache.hadoop.mapreduce.task.JobContextImpl.getWorkingDirectory(), which
is known to be non-null in
org.apache.hadoop.mapreduce.JobSubmitter.addLog4jToDistributedCache(Job, Path)
RCN Redundant nullcheck of
org.apache.hadoop.mapreduce.task.JobContextImpl.getWorkingDirectory(), which
is known to be non-null in
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(Job, Path)
RCN Redundant nullcheck of value, which is known to be non-null in
org.apache.hadoop.mapreduce.util.ResourceBundles.getValue(String, String,
String, Object)
UrF Unread public/protected field:
org.apache.hadoop.mapred.lib.CombineFileRecordReader.rrClass
UrF Unread public/protected field:
org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.rrClass
UuF Unused public or protected field:
org.apache.hadoop.mapred.lib.CombineFileRecordReader.fs
UuF Unused public or protected field:
org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader.fs
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) TestHighRamJob fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Attachment: MAPREDUCE-6234.001.patch
TestHighRamJob fails
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) TestHighRamJob fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301947#comment-14301947
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
The cause of the test failure is the change of default value of
{{mapreduce.map.memory.mb}} and {{mapreduce.reduce.memory.mb}} by
MAPREDUCE-5785. Though the default value defined in mapred-default.xml from
1024 to -1 (which means turning on auto deriving from the value of -Xmx in
java.opts), {{MRJobConfig.DEFAULT_MAP_MEMORY_MB}} and
{{MRJobConfig.DEFAULT_REDUCE_MEMORY_MB}} are still 1024. Those are used in a
few place such as the test in Gridmix.
TestHighRamJob fails
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) TestHighRamJob fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Description:
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
was:
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
TestHighRamJob fails
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6234:
Summary: MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with
mapred-default.xml (was: TestHighRamJob fails)
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302279#comment-14302279
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
Findbugs warnings are irrelevant. There is work in progress in MAPREDUCE-6225.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6234) MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
[
https://issues.apache.org/jira/browse/MAPREDUCE-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302280#comment-14302280
]
Masatake Iwasaki commented on MAPREDUCE-6234:
-
I did not add tests because the existing tests in TestJobConf cover the change.
MRJobConfig.DEFAULT_*_MEMORY_MB should be consistent with mapred-default.xml
Key: MAPREDUCE-6234
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6234
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix, mrv2
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Attachments: MAPREDUCE-6234.001.patch
TestHighRamJob fails by this.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapred.gridmix.TestHighRamJob
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.162 sec
FAILURE! - in org.apache.hadoop.mapred.gridmix.TestHighRamJob
testHighRamFeatureEmulation(org.apache.hadoop.mapred.gridmix.TestHighRamJob)
Time elapsed: 1.102 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamConfig(TestHighRamJob.java:98)
at
org.apache.hadoop.mapred.gridmix.TestHighRamJob.testHighRamFeatureEmulation(TestHighRamJob.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
