[jira] [Commented] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs

2017-04-20 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977310#comment-15977310
 ] 

Robert Kanter commented on MAPREDUCE-6871:
--

Test failure unrelated

> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch, 
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch, 
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when 
> submitting an AM so that you can actually do rack or node locality.  We 
> should allow MapReduce users to take advantage of this by exposing this 
> functionality in some way.  The raw YARN API allows for a lot of flexibility 
> (e.g. different resources per request, etc), but we don't necessarily want to 
> allow the user to do too much here so they don't shoot themselves in the foot 
> and we don't make this overly complicated.  
> I propose we allow users to specify racks and nodes for strict locality.  
> This would allow users to restrict an MR AM to specific racks and/or nodes.  
> We could add a new property, 
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a 
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding 
> {{ResourceRequest}}'s.  
> For example, 
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would 
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}} 
> {{ResourceRequest}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs

2017-04-20 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977272#comment-15977272
 ] 

Hadoop QA commented on MAPREDUCE-6871:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
34s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s 
{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
in trunk has 3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 47s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 9s {color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 18s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestLocalMRNotification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12864311/MAPREDUCE-6871.005.patch
 |
| JIRA Issue | MAPREDUCE-6871 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 797fc76746c3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c0cf11e |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6962/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
 |
| unit |

[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs

2017-04-20 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.005.patch

I was talking to [~templedf] about regexes and he suggested using 
{{String.format}} to make the regex as one String, which makes it easier to 
read.  The 005 patch does that.

> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch, 
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch, 
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when 
> submitting an AM so that you can actually do rack or node locality.  We 
> should allow MapReduce users to take advantage of this by exposing this 
> functionality in some way.  The raw YARN API allows for a lot of flexibility 
> (e.g. different resources per request, etc), but we don't necessarily want to 
> allow the user to do too much here so they don't shoot themselves in the foot 
> and we don't make this overly complicated.  
> I propose we allow users to specify racks and nodes for strict locality.  
> This would allow users to restrict an MR AM to specific racks and/or nodes.  
> We could add a new property, 
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a 
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding 
> {{ResourceRequest}}'s.  
> For example, 
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would 
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}} 
> {{ResourceRequest}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6871) Allow users to specify racks and nodes for strict locality for AMs

2017-04-20 Thread Robert Kanter (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated MAPREDUCE-6871:
-
Attachment: MAPREDUCE-6871.005.patch

I was talking to [~templedf] about regexes and he suggested using 
{{String.format}} to make the regex as one String, which makes it easier to 
read.  The 005 patch does that.

> Allow users to specify racks and nodes for strict locality for AMs
> --
>
> Key: MAPREDUCE-6871
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6871
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>  Components: client
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-6871.001.patch, MAPREDUCE-6871.002.patch, 
> MAPREDUCE-6871.003.patch, MAPREDUCE-6871.004.patch, MAPREDUCE-6871.005.patch, 
> MAPREDUCE-6871.005.patch
>
>
> YARN-6050 fixed the YARN API to allow multiple {{ResourceRequest}}'s when 
> submitting an AM so that you can actually do rack or node locality.  We 
> should allow MapReduce users to take advantage of this by exposing this 
> functionality in some way.  The raw YARN API allows for a lot of flexibility 
> (e.g. different resources per request, etc), but we don't necessarily want to 
> allow the user to do too much here so they don't shoot themselves in the foot 
> and we don't make this overly complicated.  
> I propose we allow users to specify racks and nodes for strict locality.  
> This would allow users to restrict an MR AM to specific racks and/or nodes.  
> We could add a new property, 
> {{mapreduce.job.am.resource-request.strict.locality}}, which takes a 
> comma-separated list of entries like:
> - {{/}}
> - {{//}}
> - {{}} (assumes /default-rack)
> MapReduce would then use this information to create the corresponding 
> {{ResourceRequest}}'s.  
> For example, 
> {{mapreduce.job.am.resource-request.strict.locality=/rack1/node1}} would 
> create the following {{ResourceRequest}}'s:
> - resourceName=ANY, relaxLocality=false, capability=
> - resourceName=/rack1, relaxLocality=false, capability=
> - resourceName=node1, relaxLocality=true, capability=
> By default, the property would be unset, and you'd get the normal {{ANY}} 
> {{ResourceRequest}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6879) TestDFSIO#sequentialTest throws java.lang.NullPointerException due to uninitialized IOStream

2017-04-20 Thread vincent he (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976834#comment-15976834
 ] 

vincent he commented on MAPREDUCE-6879:
---

The patch is available, please check.

> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized IOStream
> 
>
> Key: MAPREDUCE-6879
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6879
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: benchmarks
>Affects Versions: 2.7.3, 3.0.0-alpha2
>Reporter: vincent he
>Priority: Minor
>
> When I use {color:red} -seq {color} arg to write files, 
> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized stream inherited from IOMapperBase:
> {code}
> [root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
>  TestDFSIO -seq -write
> 2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:427)
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:395)
>   at org.apache.hadoop.fs.TestDFSIO.sequentialTest(TestDFSIO.java:722)
>   at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:846)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:731)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at 
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
>   at 
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6879) TestDFSIO#sequentialTest throws java.lang.NullPointerException due to uninitialized IOStream

2017-04-20 Thread vincent he (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

vincent he updated MAPREDUCE-6879:
--
Affects Version/s: 2.7.3
   Status: Patch Available  (was: Open)

> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized IOStream
> 
>
> Key: MAPREDUCE-6879
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6879
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: benchmarks
>Affects Versions: 3.0.0-alpha2, 2.7.3
>Reporter: vincent he
>Priority: Minor
>
> When I use {color:red} -seq {color} arg to write files, 
> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized stream inherited from IOMapperBase:
> {code}
> [root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
>  TestDFSIO -seq -write
> 2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:427)
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:395)
>   at org.apache.hadoop.fs.TestDFSIO.sequentialTest(TestDFSIO.java:722)
>   at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:846)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:731)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at 
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
>   at 
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6879) TestDFSIO#sequentialTest throws java.lang.NullPointerException due to uninitialized IOStream

2017-04-20 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976317#comment-15976317
 ] 

ASF GitHub Bot commented on MAPREDUCE-6879:
---

GitHub user wenxinhe opened a pull request:

https://github.com/apache/hadoop/pull/215

MAPREDUCE-6879. TestDFSIO#sequentialTest throws 
java.lang.NullPointerException due to uninitialized IOStream


[https://issues.apache.org/jira/browse/MAPREDUCE-6879](https://issues.apache.org/jira/browse/MAPREDUCE-6879)

When I use -seq arg to write files, TestDFSIO#sequentialTest throws 
java.lang.NullPointerException due to uninitialized stream inherited from 
IOMapperBase:

[root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
 TestDFSIO -seq -write
2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
java.lang.NullPointerException
at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:427)
at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:395)
at org.apache.hadoop.fs.TestDFSIO.sequentialTest(TestDFSIO.java:722)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:846)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at 
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
at 
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wenxinhe/hadoop trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #215


commit 4d7caeffc54cc5f74ed2f28f262f86ef2c77cfed
Author: vincent he 
Date:   2017-04-20T08:41:05Z

MAPREDUCE-6879. TestDFSIO#sequentialTest throws 
java.lang.NullPointerException due to uninitialized IOStream




> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized IOStream
> 
>
> Key: MAPREDUCE-6879
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6879
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: benchmarks
>Affects Versions: 3.0.0-alpha2
>Reporter: vincent he
>Priority: Minor
>
> When I use {color:red} -seq {color} arg to write files, 
> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized stream inherited from IOMapperBase:
> {code}
> [root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
>  TestDFSIO -seq -write
> 2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
>

[jira] [Commented] (MAPREDUCE-6879) TestDFSIO#sequentialTest throws java.lang.NullPointerException due to uninitialized IOStream

2017-04-20 Thread vincent he (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976312#comment-15976312
 ] 

vincent he commented on MAPREDUCE-6879:
---

The bug may be introduced from this 
[commit|https://github.com/apache/hadoop/commit/1aba4503cfc8aaa67c42179f7a69483deb3cf8d9].
  I am trying to fix it. Please assign to me.

> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized IOStream
> 
>
> Key: MAPREDUCE-6879
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6879
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: benchmarks
>Affects Versions: 3.0.0-alpha2
>Reporter: vincent he
>Priority: Minor
>
> When I use {color:red} -seq {color} arg to write files, 
> TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
> uninitialized stream inherited from IOMapperBase:
> {code}
> [root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
> share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
>  TestDFSIO -seq -write
> 2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
> 2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
> 2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> java.lang.NullPointerException
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:427)
>   at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:395)
>   at org.apache.hadoop.fs.TestDFSIO.sequentialTest(TestDFSIO.java:722)
>   at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:846)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
>   at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:731)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>   at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>   at 
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
>   at 
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6879) TestDFSIO#sequentialTest throws java.lang.NullPointerException due to uninitialized IOStream

2017-04-20 Thread vincent he (JIRA) 
vincent he created MAPREDUCE-6879:
-

 Summary: TestDFSIO#sequentialTest throws 
java.lang.NullPointerException due to uninitialized IOStream
 Key: MAPREDUCE-6879
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6879
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: benchmarks
Affects Versions: 3.0.0-alpha2
Reporter: vincent he
Priority: Minor


When I use {color:red} -seq {color} arg to write files, 
TestDFSIO#sequentialTest throws java.lang.NullPointerException due to 
uninitialized stream inherited from IOMapperBase:

{code}
[root@7b65290f2609 hadoop-3.0.0-alpha3-SNAPSHOT]# bin/hadoop jar 
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar
 TestDFSIO -seq -write
2017-04-20 15:50:43,505 INFO fs.TestDFSIO: TestDFSIO.1.8
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrFiles = 1
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: bufferSize = 100
2017-04-20 15:50:43,510 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2017-04-20 15:50:43,655 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
java.lang.NullPointerException
at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:427)
at org.apache.hadoop.fs.TestDFSIO$WriteMapper.doIO(TestDFSIO.java:395)
at org.apache.hadoop.fs.TestDFSIO.sequentialTest(TestDFSIO.java:722)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:846)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:731)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at 
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
at 
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org