[jira] [Commented] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Haibo Chen (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326720#comment-15326720
 ] 

Haibo Chen commented on MAPREDUCE-6643:
---

Tested the patch against some other seeds with which I have seen the test 
failed. All green after applying the patch.

> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
> ---
>
> Key: MAPREDUCE-6643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Kai Sasaki
> Attachments: MAPREDUCE-6643.01.patch, MAPREDUCE-6643.02.patch, 
> MAPREDUCE-6643.03.patch
>
>
> Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is  
> 1313094493.
> Stacktrace
> java.lang.AssertionError: Key in multiple partitions.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at 
> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326554#comment-15326554
 ] 

Akira AJISAKA edited comment on MAPREDUCE-6643 at 6/12/16 5:38 PM:
---

Thanks [~lewuathe] for updating the patch! Mostly looks good to me. Some 
comments:
{code}
if (start == 0) {
  adjStart = Math.max(0L, start - FIRST_BZIP2_BLOCK_MARKER_POSITION);
{code}
1. It seems that adjStart is always zero. If that is true, we should set zero 
instead of using max function.

{code}
  // Other than the fist of file, the marker size is 6 bytes.
{code}
2. fist -> first
3. Would you fix the checkstyle warning?
{code}
// corner case when we have byte alignment and position of stream are same
int[] lengths = {471507, 473608};
int[] numSplits = {218, 110};

for (int i = 0; i < 2; i++) {
  verifyPartitions(lengths[i], numSplits[i], file, codec, conf);
}
{code}
4. (nit) The following code is simpler for me:
{code}
// corner case when we have byte alignment and position of stream are same
verifyPartitions(471507, 218, file, codec, conf);
verifyPartitions(473608, 110, file, codec, conf);
{code}


was (Author: ajisakaa):
Thanks [~lewuathe] for updating the patch! Mostly looks good to me. Some 
comments:
{code}
if (start == 0) {
  adjStart = Math.max(0L, start - FIRST_BZIP2_BLOCK_MARKER_POSITION);
{code}
1. It seems that adjStart is always zero. If that is true, we should set zero 
instead of using max function.

{code}
  // Other than the fist of file, the marker size is 6 bytes.
{code}
2. fist -> first
3. Would you fix the checkstyle warning?
{code}
// corner case when we have byte alignment and position of stream are same
int[] lengths = {471507, 473608};
int[] numSplits = {218, 110};

for (int i = 0; i < 2; i++) {
  verifyPartitions(lengths[i], numSplits[i], file, codec, conf);
}
{code}
4. (nit) The following code is simpler for me:
{code}
// corner case when we have byte alignment and position of stream are same
verifyPartition(471507, 218, file, codec, conf);
verifyPartition(473608, 110, file, codec, conf);
{code}

> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
> ---
>
> Key: MAPREDUCE-6643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Kai Sasaki
> Attachments: MAPREDUCE-6643.01.patch, MAPREDUCE-6643.02.patch, 
> MAPREDUCE-6643.03.patch
>
>
> Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is  
> 1313094493.
> Stacktrace
> java.lang.AssertionError: Key in multiple partitions.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at 
> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Akira AJISAKA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326554#comment-15326554
 ] 

Akira AJISAKA commented on MAPREDUCE-6643:
--

Thanks [~lewuathe] for updating the patch! Mostly looks good to me. Some 
comments:
{code}
if (start == 0) {
  adjStart = Math.max(0L, start - FIRST_BZIP2_BLOCK_MARKER_POSITION);
{code}
1. It seems that adjStart is always zero. If that is true, we should set zero 
instead of using max function.

{code}
  // Other than the fist of file, the marker size is 6 bytes.
{code}
2. fist -> first
3. Would you fix the checkstyle warning?
{code}
// corner case when we have byte alignment and position of stream are same
int[] lengths = {471507, 473608};
int[] numSplits = {218, 110};

for (int i = 0; i < 2; i++) {
  verifyPartitions(lengths[i], numSplits[i], file, codec, conf);
}
{code}
4. (nit) The following code is simpler for me:
{code}
// corner case when we have byte alignment and position of stream are same
verifyPartition(471507, 218, file, codec, conf);
verifyPartition(473608, 110, file, codec, conf);
{code}

> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
> ---
>
> Key: MAPREDUCE-6643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Kai Sasaki
> Attachments: MAPREDUCE-6643.01.patch, MAPREDUCE-6643.02.patch, 
> MAPREDUCE-6643.03.patch
>
>
> Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is  
> 1313094493.
> Stacktrace
> java.lang.AssertionError: Key in multiple partitions.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at 
> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326467#comment-15326467
 ] 

Hadoop QA commented on MAPREDUCE-6643:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 41s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 20s 
{color} | {color:red} root: The patch generated 1 new + 54 unchanged - 2 fixed 
= 55 total (was 56) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 20 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 30s {color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 37s 
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 53s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMRCJCFileOutputCommitter |
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12809696/MAPREDUCE-6643.03.patch
 |
| JIRA Issue | MAPREDUCE-6643 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 72dc0063f6ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 5279af7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/diff-checkstyle-root.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6549/artifact/patchprocess/whitespace-eol.txt
 |
| unit | 
https://builds.apache.

[jira] [Commented] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Kai Sasaki (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326400#comment-15326400
 ] 

Kai Sasaki commented on MAPREDUCE-6643:
---

I found {{BZip2Codec}} always back 10bytes to search compression marker even 
markers except for first marker has 6bytes. It can let 
{{BZip2CompressionInputStream}} find same marker twice because the search start 
points is back more than marker size. I fixed compression stream only back 
6bytes other than the file start point.

> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
> ---
>
> Key: MAPREDUCE-6643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Kai Sasaki
> Attachments: MAPREDUCE-6643.01.patch, MAPREDUCE-6643.02.patch, 
> MAPREDUCE-6643.03.patch
>
>
> Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is  
> 1313094493.
> Stacktrace
> java.lang.AssertionError: Key in multiple partitions.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at 
> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-6643) org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed

2016-06-12 Thread Kai Sasaki (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Sasaki updated MAPREDUCE-6643:
--
Attachment: MAPREDUCE-6643.03.patch

> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs failed
> ---
>
> Key: MAPREDUCE-6643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6643
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Haibo Chen
>Assignee: Kai Sasaki
> Attachments: MAPREDUCE-6643.01.patch, MAPREDUCE-6643.02.patch, 
> MAPREDUCE-6643.03.patch
>
>
> Unit test TestTextInputFormat.testSplitableCodecs() failed when the seed is  
> 1313094493.
> Stacktrace
> java.lang.AssertionError: Key in multiple partitions.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at 
> org.apache.hadoop.mapred.TestTextInputFormat.testSplitableCodecs(TestTextInputFormat.java:223)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org