[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248278#comment-16248278 ] Hudson commented on HADOOP-8522: SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13222 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13222/]) HADOOP-8522. ResetableGzipOutputStream creates invalid gzip files when (cdouglas: rev 796a0d3a5c661f0c3b23af9c0db2d8f3db83c322) * (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestGzipCodec.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/GzipCodec.java > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Fix For: 3.0.0 > > Attachments: HADOOP-8522-4.patch, HADOOP-8522.05.patch, > HADOOP-8522.06.patch, HADOOP-8522.07.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248229#comment-16248229 ] Chris Douglas commented on HADOOP-8522: --- If there's no other feedback, I'll commit this. > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Attachments: HADOOP-8522-4.patch, HADOOP-8522.05.patch, > HADOOP-8522.06.patch, HADOOP-8522.07.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246720#comment-16246720
]
Chris Douglas commented on HADOOP-8522:
---
bq. One thing that caught my eye was the decision to make the methods out the
GZipOutputStream synchronized
The supertype ({{GZIPOutputStream}}) synchronizes these methods, so the patch
synchronizes consistently with it.
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used
>
>
> Key: HADOOP-8522
> URL: https://issues.apache.org/jira/browse/HADOOP-8522
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 1.0.3, 2.0.0-alpha
>Reporter: Mike Percy
>Assignee: Mike Percy
> Labels: BB2015-05-TBR
> Attachments: HADOOP-8522-4.patch, HADOOP-8522.05.patch,
> HADOOP-8522.06.patch, HADOOP-8522.07.patch
>
>
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used. The issue is that finish() flushes the compressor
> buffer and writes the gzip CRC32 + data length trailer. After that,
> resetState() does not repeat the gzip header, but simply starts writing more
> deflate-compressed data. The resultant files are not readable by the Linux
> "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip
> files.
> The gzip format is specified in [RFC
> 1952|https://tools.ietf.org/html/rfc1952].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241410#comment-16241410 ] Mike Percy commented on HADOOP-8522: [~chris.douglas] Overall the changes look good to me. [~fabbri] do you have any additional thoughts on this? You took a look at rev 4 from me some time back. One thing that caught my eye was the decision to make the methods out the GZipOutputStream synchronized. If the contract for CompressionOutputStream is not thread safe it seems like overkill. > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Attachments: HADOOP-8522-4.patch, HADOOP-8522.05.patch, > HADOOP-8522.06.patch, HADOOP-8522.07.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241311#comment-16241311
]
Hadoop QA commented on HADOOP-8522:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 48s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 16s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-8522 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12896292/HADOOP-8522.07.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 993b1bd675cc 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8f214dc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13638/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13638/testReport/ |
| Max. process+thread count | 1336 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13638/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240989#comment-16240989
]
Hadoop QA commented on HADOOP-8522:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
20s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red} hadoop-common in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 6 new + 8 unchanged - 0 fixed = 14 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
40s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 44s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 62 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-common:1 |
| | hadoop-common:4 |
| Failed junit tests | hadoop.ipc.TestCallQueueManager |
| | hadoop.security.TestUserGroupInformation |
| | hadoop.http.TestHttpServer |
| | hadoop.security.token.delegation.web.TestWebDelegationToken |
| | hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
| | hadoop.ipc.TestIPCServerResponder |
| | hadoop.ipc.TestIPC |
| | hadoop.log.TestLogLevel |
| | hadoop.ha.TestZKFailoverControllerStress |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-8522 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12896261/HADOOP-8522.06.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c9ba03b720fc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / dcd99c4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs |
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892879#comment-15892879 ] Mike Percy commented on HADOOP-8522: I got some review feedback on this offline, I think the patch needs to be updated again. This was the feedback: Why does resetState() write a new header to the stream.. versus, say, doing it lazily if and when more data is written? > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Attachments: HADOOP-8522-4.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890513#comment-15890513
]
Hadoop QA commented on HADOOP-8522:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red} HADOOP-8522 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-8522 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822675/HADOOP-8522-4.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11740/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used
>
>
> Key: HADOOP-8522
> URL: https://issues.apache.org/jira/browse/HADOOP-8522
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 1.0.3, 2.0.0-alpha
>Reporter: Mike Percy
>Assignee: Mike Percy
> Labels: BB2015-05-TBR
> Attachments: HADOOP-8522-4.patch
>
>
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used. The issue is that finish() flushes the compressor
> buffer and writes the gzip CRC32 + data length trailer. After that,
> resetState() does not repeat the gzip header, but simply starts writing more
> deflate-compressed data. The resultant files are not readable by the Linux
> "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip
> files.
> The gzip format is specified in [RFC
> 1952|https://tools.ietf.org/html/rfc1952].
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890492#comment-15890492 ] Narayan Periwal commented on HADOOP-8522: - I am also facing the same issue of the invalid gzip files. Will appreciate if this can be merged. > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Attachments: HADOOP-8522-4.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429929#comment-15429929 ] Mike Percy commented on HADOOP-8522: Ping on this JIRA. Can this be committed? > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used > > > Key: HADOOP-8522 > URL: https://issues.apache.org/jira/browse/HADOOP-8522 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Mike Percy >Assignee: Mike Percy > Labels: BB2015-05-TBR > Attachments: HADOOP-8522-4.patch > > > ResetableGzipOutputStream creates invalid gzip files when finish() and > resetState() are used. The issue is that finish() flushes the compressor > buffer and writes the gzip CRC32 + data length trailer. After that, > resetState() does not repeat the gzip header, but simply starts writing more > deflate-compressed data. The resultant files are not readable by the Linux > "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip > files. > The gzip format is specified in [RFC > 1952|https://tools.ietf.org/html/rfc1952]. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412637#comment-15412637
]
Hadoop QA commented on HADOOP-8522:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 7 new + 10 unchanged - 1 fixed = 17 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
17s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822675/HADOOP-8522-4.patch |
| JIRA Issue | HADOOP-8522 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1b19f92eb870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0705489 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10204/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10204/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10204/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used
>
>
> Key: HADOOP-8522
> URL: https://issues.apache.org/jira/browse/HADOOP-8522
> Project: Hadoop Common
> Issue Type: Bug
>
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410302#comment-15410302
]
Hadoop QA commented on HADOOP-8522:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 5 new + 10 unchanged - 1 fixed = 15 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 43s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.io.compress.TestGzipCodec |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822397/HADOOP-8522-3.patch |
| JIRA Issue | HADOOP-8522 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1816b1c701fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d81b816 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10190/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10190/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10190/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10190/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used
>
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410128#comment-15410128
]
Hadoop QA commented on HADOOP-8522:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-8522 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12532858/HADOOP-8522-2a.patch |
| JIRA Issue | HADOOP-8522 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10189/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used
>
>
> Key: HADOOP-8522
> URL: https://issues.apache.org/jira/browse/HADOOP-8522
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 1.0.3, 2.0.0-alpha
>Reporter: Mike Percy
>Assignee: Mike Percy
> Labels: BB2015-05-TBR
> Attachments: HADOOP-8522-2a.patch
>
>
> ResetableGzipOutputStream creates invalid gzip files when finish() and
> resetState() are used. The issue is that finish() flushes the compressor
> buffer and writes the gzip CRC32 + data length trailer. After that,
> resetState() does not repeat the gzip header, but simply starts writing more
> deflate-compressed data. The resultant files are not readable by the Linux
> "gunzip" tool. ResetableGzipOutputStream should write valid multi-member gzip
> files.
> The gzip format is specified in [RFC
> 1952|https://tools.ietf.org/html/rfc1952].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[
https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524555#comment-14524555
]
Hadoop QA commented on HADOOP-8522:
---
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12532858/HADOOP-8522-2a.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/6256/console |
This message was automatically generated.
ResetableGzipOutputStream creates invalid gzip files when finish() and
resetState() are used
Key: HADOOP-8522
URL: https://issues.apache.org/jira/browse/HADOOP-8522
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Mike Percy
Assignee: Mike Percy
Attachments: HADOOP-8522-2a.patch
ResetableGzipOutputStream creates invalid gzip files when finish() and
resetState() are used. The issue is that finish() flushes the compressor
buffer and writes the gzip CRC32 + data length trailer. After that,
resetState() does not repeat the gzip header, but simply starts writing more
deflate-compressed data. The resultant files are not readable by the Linux
gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip
files.
The gzip format is specified in [RFC
1952|https://tools.ietf.org/html/rfc1952].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422904#comment-13422904 ] Mike Percy commented on HADOOP-8522: Sincere apologies, but I just don't see myself finding the time to improve this unit test very soon. I have a lot going on over in Flume land. Would it be alright if we file another JIRA to improve the unit test and move forward with committing this patch? Just in case that is OK, I have filed HADOOP-8625 for that. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Assignee: Mike Percy Attachments: HADOOP-8522-2a.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13416395#comment-13416395 ] Tom White commented on HADOOP-8522: --- Tom, do you expect GzipCodec to be able to read multi-member gzip files on JDK1.6 without native libs installed? Yes, Hadoop's non-native BuiltInGzipDecompressor supports concatenated gzip. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Assignee: Mike Percy Attachments: HADOOP-8522-2a.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13415711#comment-13415711 ] Mike Percy commented on HADOOP-8522: Apologies for the delay in following up on this. Tom, do you expect GzipCodec to be able to read multi-member gzip files on JDK1.6 without native libs installed? ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Assignee: Mike Percy Attachments: HADOOP-8522-2a.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399224#comment-13399224 ] Mike Percy commented on HADOOP-8522: Tom, thanks for the review! I will update the patch. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Attachments: HADOOP-8522-2a.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398288#comment-13398288 ] Mike Percy commented on HADOOP-8522: Some additional info. Currently, resetState() works when using the native Zlib gzip implementation; the output appears to comply with the spec and works with gunzip because it writes the full header and trailer (basically concatenated gzip files). That may be one reason this bug has lain dormant for so long with the non-native implementation (serious users tend to use the native libs). So, the problem is with the non-native gzip implementation. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398292#comment-13398292 ] Mike Percy commented on HADOOP-8522: (note: trunk patch) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Attachments: HADOOP-8522-2.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8522) ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used
[ https://issues.apache.org/jira/browse/HADOOP-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398854#comment-13398854 ] Tom White commented on HADOOP-8522: --- The patch looks good to me. A small suggestion regarding the test: why not use GzipCodec to decompress too? Then you can have an assert in the test, and it checks roundtripping using Hadoop APIs. That may be one reason this bug has lain dormant for so long with the non-native implementation (serious users tend to use the native libs). Also, Hadoop has only supported concatenated gzip since HADOOP-6835, so files that had corrupt later members would have been ignored by versions of Hadoop prior to this. ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used Key: HADOOP-8522 URL: https://issues.apache.org/jira/browse/HADOOP-8522 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Mike Percy Attachments: HADOOP-8522-2a.patch ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used. The issue is that finish() flushes the compressor buffer and writes the gzip CRC32 + data length trailer. After that, resetState() does not repeat the gzip header, but simply starts writing more deflate-compressed data. The resultant files are not readable by the Linux gunzip tool. ResetableGzipOutputStream should write valid multi-member gzip files. The gzip format is specified in [RFC 1952|https://tools.ietf.org/html/rfc1952]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
