[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs

2016-02-09 Thread Sunil G (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140368#comment-15140368
 ] 

Sunil G commented on MAPREDUCE-6304:


cc/ [~Naganarasimha],  [~leftnoteasy],  [~djp] 

Currently in 2.6 branch,  this patch s not present. So for users who wants to 
specify label at app submission time,  this feature is not available .  I think 
we can backport this to 2.6 line. I had mentioned this point in release mail 
thread also. Thoughts?

> Specifying node labels when submitting MR jobs
> --
>
> Key: MAPREDUCE-6304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Jian Fang
>Assignee: Naganarasimha G R
>  Labels: mapreduce
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6304.20150410-1.patch, 
> MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, 
> MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch, 
> MAPREDUCE-6304.20150512-1.patch
>
>
> Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify 
> node labels when submitting MR jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6626) Reuse ObjectMapper instance in MapReduce

2016-02-09 Thread Lin Yiqun (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140229#comment-15140229
 ] 

Lin Yiqun commented on MAPREDUCE-6626:
--

Thanks [~ajisakaa] for commit and thanks [~brahmareddy] for review.

> Reuse ObjectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6630) TestUserGroupInformation#testGetServerSideGroups fails in chroot

2016-02-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139997#comment-15139997
 ] 

Hadoop QA commented on MAPREDUCE-6630:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 18s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 35s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s 
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s 
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 51s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 17s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 35s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 39s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12786857/MAPREDUCE-6630-001.patch
 |
| JIRA Issue | MAPREDUCE-6630 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux c2cef33ad395 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.s

[jira] [Updated] (MAPREDUCE-6630) TestUserGroupInformation#testGetServerSideGroups fails in chroot

2016-02-09 Thread Jason Lowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-6630:
--
Target Version/s: 2.7.3
  Status: Patch Available  (was: Open)

> TestUserGroupInformation#testGetServerSideGroups fails in chroot
> 
>
> Key: MAPREDUCE-6630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6630
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: security, test
>Affects Versions: 2.1.0-beta
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Minor
> Attachments: MAPREDUCE-6630-001.patch
>
>
> Bug fixed by [HADOOP-7811] broken by [HADOOP-8562]. Need to re-introduce the 
> fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set

2016-02-09 Thread Tsuyoshi Ozawa (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139785#comment-15139785
 ] 

Tsuyoshi Ozawa commented on MAPREDUCE-6607:
---

OK, I'll check this issue today.

> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or 
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned 
> up:
> * mapreduce.task.files.preserve.failedtask 
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter 
> was supposed to be used only if that task name matches against the specified 
> regular expression.
> {code}
>   protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
>   }
> {code}
> {code}
>   public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
>   if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
>   LOG.warn("Job Staging directory is null");
>   return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " + 
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
>   }
> } catch(IOException io) {
>   LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6579) JobStatus#getFailureInfo should not output diagnostic information when the job is running

2016-02-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139758#comment-15139758
 ] 

Hadoop QA commented on MAPREDUCE-6579:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s 
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 36s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch 
generated 4 new + 89 unchanged - 1 fixed = 93 total (was 90) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 53s 
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed 
with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 32s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed 
with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 31s 
{color} | {color:red} Patch generated 15 ASF License warnings. {co

[jira] [Commented] (MAPREDUCE-6626) Reuse ObjectMapper instance in MapReduce

2016-02-09 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139333#comment-15139333
 ] 

Hudson commented on MAPREDUCE-6626:
---

FAILURE: Integrated in Hadoop-trunk-Commit #9267 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9267/])
MAPREDUCE-6626. Reuse ObjectMapper instance in MapReduce. Contributed by 
(aajisaka: rev a0b1f10a30dc2736cc136f257b0d3bf0140158bb)
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* hadoop-mapreduce-project/CHANGES.txt
* 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java


> Reuse ObjectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6626) Reuse ObjectMapper instance in MapReduce

2016-02-09 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139317#comment-15139317
 ] 

Akira AJISAKA commented on MAPREDUCE-6626:
--

Committed this to trunk, branch-2, and branch-2.8. Thanks [~linyiqun] for 
creating patches and thanks [~brahmareddy] for the review.

> Reuse ObjectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6626) Reuse ObjectMapper instance in MapReduce

2016-02-09 Thread Akira AJISAKA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6626:
-
   Resolution: Fixed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

> Reuse ObjectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6626) Reuse ObjectMapper instance in MapReduce

2016-02-09 Thread Akira AJISAKA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6626:
-
Priority: Minor  (was: Major)
Hadoop Flags: Reviewed
 Summary: Reuse ObjectMapper instance in MapReduce  (was: Reuse 
objectMapper instance in MapReduce)

> Reuse ObjectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Minor
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce

2016-02-09 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139245#comment-15139245
 ] 

Akira AJISAKA commented on MAPREDUCE-6626:
--

LGTM, +1.

> Reuse objectMapper instance in MapReduce
> 
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE-6626.003.patch, 
> MAPREDUCE-6626.004.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance 
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of 
> instances) for data binding; many optimizations for reuse (of symbol tables, 
> some buffers) depend on ObjectMapper instances being reused. 
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
>  it's similar to HDFS-9724.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6579) JobStatus#getFailureInfo should not output diagnostic information when the job is running

2016-02-09 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139224#comment-15139224
 ] 

Akira AJISAKA commented on MAPREDUCE-6579:
--

Raised the priority to blocker since the issue is to keep API compatibility.

> JobStatus#getFailureInfo should not output diagnostic information when the 
> job is running
> -
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
>Priority: Blocker
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch, 
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch, MAPREDUCE-6579.05.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 30.55 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015] 
> Application is Activated, waiting for resources to be assigned for AM.  
> Details : AM Partition =  ; Partition Resource = 
>  ; Queue's Absolute capacity = 100.0 % ; Queue's 
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6579) JobStatus#getFailureInfo should not output diagnostic information when the job is running

2016-02-09 Thread Akira AJISAKA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6579:
-
Priority: Blocker  (was: Major)
 Summary: JobStatus#getFailureInfo should not output diagnostic information 
when the job is running  (was: Test failure : TestNetworkedJob)

> JobStatus#getFailureInfo should not output diagnostic information when the 
> job is running
> -
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
>Priority: Blocker
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch, 
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch, MAPREDUCE-6579.05.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 30.55 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015] 
> Application is Activated, waiting for resources to be assigned for AM.  
> Details : AM Partition =  ; Partition Resource = 
>  ; Queue's Absolute capacity = 100.0 % ; Queue's 
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6579) Test failure : TestNetworkedJob

2016-02-09 Thread Akira AJISAKA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6579:
-
Attachment: MAPREDUCE-6579.05.patch

> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch, 
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch, MAPREDUCE-6579.05.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 30.55 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015] 
> Application is Activated, waiting for resources to be assigned for AM.  
> Details : AM Partition =  ; Partition Resource = 
>  ; Queue's Absolute capacity = 100.0 % ; Queue's 
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6413) TestLocalJobSubmission is failing with unknown host

2016-02-09 Thread Jason Lowe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-6413:
--
Fix Version/s: 2.6.5
   2.7.3

I committed this to branch-2.7 and branch-2.6 since MAPREDUCE-6238 was also 
brought into those branches.

> TestLocalJobSubmission is failing with unknown host
> ---
>
> Key: MAPREDUCE-6413
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6413
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Affects Versions: 2.8.0
>Reporter: Jason Lowe
>Assignee: zhihai xu
> Fix For: 2.8.0, 2.7.3, 2.6.5
>
> Attachments: MAPREDUCE-6413.000.patch, MAPREDUCE-6413.001.patch, 
> MAPREDUCE-6413.002.patch
>
>
> ThestLocalJobSubmission.testLocalJobLibjarsOption is failing with 
> java.net.UnknownHostException: testcluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob

2016-02-09 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139173#comment-15139173
 ] 

Akira AJISAKA commented on MAPREDUCE-6579:
--

bq. One approach i can think of is checking the status of job in the MR client 
side and only if its failed we can fetch and return the diagnosis message from 
yarn else we can return empty string. This will ensure the compatability is not 
broken if any client application is checking the failure message with out 
checking the job status.
Sounds good. I'll update the patch. I'll update and the summary of this issue 
as well.

> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch, 
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From 
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
>  TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec 
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time elapsed: 
> 30.55 sec  <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015] 
> Application is Activated, waiting for resources to be assigned for AM.  
> Details : AM Partition =  ; Partition Resource = 
>  ; Queue's Absolute capacity = 100.0 % ; Queue's 
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> 
> but was:<[]>
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set

2016-02-09 Thread Kai Sasaki (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139004#comment-15139004
 ] 

Kai Sasaki commented on MAPREDUCE-6607:
---

[~ozawa] Sorry for bothering you but could you review this?

> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or 
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned 
> up:
> * mapreduce.task.files.preserve.failedtask 
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter 
> was supposed to be used only if that task name matches against the specified 
> regular expression.
> {code}
>   protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
>   }
> {code}
> {code}
>   public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
>   if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
>   LOG.warn("Job Staging directory is null");
>   return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " + 
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
>   }
> } catch(IOException io) {
>   LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)