[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299783#comment-17299783 ] Steve Loughran commented on HADOOP-15870: - where are we with this to get it into 3.3.1? > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952432#comment-16952432 ] lqjacklee commented on HADOOP-15870: HADOOP-15870-006.patch provide the option `support-available-is-zero` to check the available is zero support provide the option `support-available-is-positive` to check the available is positive support provide the option `support-available-at-eof` to check the available at eof support > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch, HADOOP-15870-006.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951109#comment-16951109 ] Hudson commented on HADOOP-15870: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17531 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17531/]) Revert "HADOOP-15870. S3AInputStream.remainingInFile should use (stevel: rev dee9e97075e67f53d033df522372064ca19d6b51) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951099#comment-16951099 ] Steve Loughran commented on HADOOP-15870: - Okay I reverted the patch. We can decide what to do at our leisure. I'm thinking we may need both of * Fix WebHDFSInputStream.available() * Allow FS contracts to skip those probes (for downstream uses) > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951089#comment-16951089 ] Steve Loughran commented on HADOOP-15870: - FWIW this is showing that WebHDFSInputStream.available() is always 0. To be purist, it should be forwarding the probe all the way to the input stream. So after a revert we could actually fix that > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950806#comment-16950806 ] lqjacklee commented on HADOOP-15870: [~ayushtkn] Thank you very much. We will check it. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950795#comment-16950795
]
Ayush Saxena commented on HADOOP-15870:
---
Hi [~ste...@apache.org] [~Jack-Lee]
Seems this change broke {{TestRouterWebHDFSContractSeek}}
Can you give a check once...
Ref :
https://builds.apache.org/job/PreCommit-HDFS-Build/28085/testReport/org.apache.hadoop.fs.contract.router.web/TestRouterWebHDFSContractSeek/
> S3AInputStream.remainingInFile should use nextReadPos
> -
>
> Key: HADOOP-15870
> URL: https://issues.apache.org/jira/browse/HADOOP-15870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4, 3.1.1
>Reporter: Shixiong Zhu
>Assignee: lqjacklee
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch,
> HADOOP-15870-004.patch, HADOOP-15870-005.patch
>
>
> Otherwise `remainingInFile` will not change after `seek`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949004#comment-16949004 ] Hudson commented on HADOOP-15870: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17521 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17521/]) HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos. (stevel: rev 7a4b3d42c4e36e468c2a46fd48036a6fed547853) * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractSeek.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943506#comment-16943506
]
Hadoop QA commented on HADOOP-15870:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red} HADOOP-15870 does not apply to branch-3.2. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15870 |
| GITHUB PR | https://github.com/apache/hadoop/pull/433 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16568/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> S3AInputStream.remainingInFile should use nextReadPos
> -
>
> Key: HADOOP-15870
> URL: https://issues.apache.org/jira/browse/HADOOP-15870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4, 3.1.1
>Reporter: Shixiong Zhu
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch,
> HADOOP-15870-004.patch, HADOOP-15870-005.patch
>
>
> Otherwise `remainingInFile` will not change after `seek`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798301#comment-16798301 ] Steve Loughran commented on HADOOP-15870: - this patch should be ready to get in. Can everyone interested review and retest it? thanks > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771938#comment-16771938
]
Hadoop QA commented on HADOOP-15870:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
54s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
26s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
48s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 35s{color} | {color:orange} root: The patch generated 3 new + 10 unchanged -
0 fixed = 13 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 41s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HADOOP-15870 |
| GITHUB PR | https://github.com/apache/hadoop/pull/433 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux a9d30ffbb659 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / a060e8c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771833#comment-16771833 ] Steve Loughran commented on HADOOP-15870: - patch 005 * Clarify the Gzip bug better in the markdown * remove the this. prefix on field/method references in the changed lines run the HDFS, Azure wasb & abfs distcp tests and *all* s3a tests : all were happy > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch, HADOOP-15870-005.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771400#comment-16771400
]
Hadoop QA commented on HADOOP-15870:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
41s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
45s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 24s{color} | {color:orange} root: The patch generated 3 new + 10 unchanged -
0 fixed = 13 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HADOOP-15870 |
| GITHUB PR | https://github.com/apache/hadoop/pull/433 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8144ec5efa03 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / a060e8c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771329#comment-16771329 ] Steve Loughran commented on HADOOP-15870: - HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos patch 004 Takes patch 003, applies to trunk and then * specified what availalble should do, highlighting where GZip's expectations diverge from the Java API requirements * Seek test has specific available tests factored for better readability and reporting * S3A input stream semantics become: "return 0 on closed/past end of file; 1 when no stream is open, else wrappedStream.available() The previous patch was slightly flawed in that reading a file > 2^32 bytes long then MAX_INT would be returned, rather than the avaialable set) > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch, > HADOOP-15870-004.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711522#comment-16711522 ] lqjacklee commented on HADOOP-15870: [^HADOOP-15870-003.patch] > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch, HADOOP-15870-003.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711490#comment-16711490 ] lqjacklee commented on HADOOP-15870: [^HADOOP-15870-3.patch] > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697727#comment-16697727 ] lqjacklee commented on HADOOP-15870: https://github.com/apache/hadoop/pull/433.patch > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697567#comment-16697567 ] lqjacklee commented on HADOOP-15870: [~sunilg] [~ste...@apache.org] Thanks the comment, I have updated the patch. however I cannot upload it. Could you help that, thanks. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696028#comment-16696028 ] lqjacklee commented on HADOOP-15870: [~ste...@apache.org] How can I upload the attachment of the patch? > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694773#comment-16694773
]
Steve Loughran commented on HADOOP-15870:
-
Now, code review
{{InputStream.available()}} declares the #of bytes which can be read *without
blocking*. This is not what is currently being calculated for S3A input stream
-it's broken. And this patch, while making it more accurate, isn't doing it
either.
We should just be doing something like
{code}
public synchronized int available() throws IOException {
checkNotClosed();
return wrappedStream != null? wrappedStream.available() : 0;
}
{code}
All the new tests will validate this behaviour
> S3AInputStream.remainingInFile should use nextReadPos
> -
>
> Key: HADOOP-15870
> URL: https://issues.apache.org/jira/browse/HADOOP-15870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4, 3.1.1
>Reporter: Shixiong Zhu
>Assignee: lqjacklee
>Priority: Major
> Attachments: HADOOP-15870-002.patch
>
>
> Otherwise `remainingInFile` will not change after `seek`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693742#comment-16693742 ] Steve Loughran commented on HADOOP-15870: - Afraid you'll need to attach it to HADOOP-15920, because once yetus sees a github PR it kicks into github mode, but it doesn't quite do what we need after doing that, cancel the patch there and hit "submit patch" Again; the act of resubmitting the patch triggers the rerun > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693229#comment-16693229 ] lqjacklee commented on HADOOP-15870: [^HADOOP-15870-002.patch] > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > Attachments: HADOOP-15870-002.patch > > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[
https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691900#comment-16691900
]
Steve Loughran commented on HADOOP-15870:
-
thanks for the test run.
For the patch, use git diff:
{code}
git diff trunk...HEAD > HADOOP-15870-002.patch
{code}
Attach file to the other JIRA, hit "cancel patch" then "submit patch"
thanks
> S3AInputStream.remainingInFile should use nextReadPos
> -
>
> Key: HADOOP-15870
> URL: https://issues.apache.org/jira/browse/HADOOP-15870
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.4, 3.1.1
>Reporter: Shixiong Zhu
>Assignee: lqjacklee
>Priority: Major
>
> Otherwise `remainingInFile` will not change after `seek`.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691871#comment-16691871 ] lqjacklee commented on HADOOP-15870: [~ste...@apache.org] I verify the function under the ap-southeast-1 region. What's more, how can I attach the patch file, please help me, thanks a lot. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690813#comment-16690813 ] lqjacklee commented on HADOOP-15870: /Users/liu/java/workspace-home/git/hadoop/patch/HADOOP-15870.patch > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690632#comment-16690632 ] ASF GitHub Bot commented on HADOOP-15870: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/433 no worries > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690217#comment-16690217 ] ASF GitHub Bot commented on HADOOP-15870: - Github user lqjack commented on the issue: https://github.com/apache/hadoop/pull/433 @steveloughran Due to the upgrade to the latest version of the Mac OS, I need take some time to resolve the environment before I can run the test successful . > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685947#comment-16685947 ] lqjacklee commented on HADOOP-15870: [~ste...@apache.org] Thanks , I will update it. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685186#comment-16685186 ] Steve Loughran commented on HADOOP-15870: - Reviewing patch 001 on HADOOP-15920 Looks like the new test fails on the local tests * the assert should list the current (failing) value * we're into the corners of the semantics of "what happens at the EOF". this is a troublespot as it shows up those places where HDFS and Posix diverge. This also means we'll have to submit this patch to HDFS too, to get its test coverage. (ideally: run them locally first, so that'll just be due diligence) > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Assignee: lqjacklee >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674021#comment-16674021 ] ASF GitHub Bot commented on HADOOP-15870: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/433 As the usual test police, which S3 endpoint did you rerun all the hadoop-aws tests against? https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html This only takes < 10 minutes, even remotely ``` mvn verify -Dparallel-tests -DtestsThreadCount=10 -Dscale # or with s3guard mvn verify -Dparallel-tests -DtestsThreadCount=10 -Ds3guard -Ddynamodb -Dscale ``` I'll help with testing the other stores (azure wasb/abfs/adl), before the commit, but I'd appreciate the diligence here as it shows you've done the homework. thx > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663065#comment-16663065 ] ASF GitHub Bot commented on HADOOP-15870: - Github user lqjack commented on the issue: https://github.com/apache/hadoop/pull/433 @steveloughran I have updated the code , please help reveiw and comment, thanks . > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661733#comment-16661733 ] ASF GitHub Bot commented on HADOOP-15870: - Github user lqjack commented on the issue: https://github.com/apache/hadoop/pull/433 @steveloughran Thanks very much , I will update. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660784#comment-16660784 ] ASF GitHub Bot commented on HADOOP-15870: - Github user steveloughran commented on the issue: https://github.com/apache/hadoop/pull/433 1. I think the same problem surfaces in `remainingInCurrentRequest()` too, which is where some inspection is going to be needed to carefully understand what's up. 1. For this method, I think maybe we could cut it completely, and fix up `available()` to return a default value when `wrappedStream==null` (0 or maybe 1 except when at EOF), and when the stream isn't null, delegate to it. That pushes down the problem of estimating the number of non-blocking bytes into the http connector. Tests. In `AbstractContractSeekTest` there's some existing tests which could take some more asserts on that available call * `testSeekZeroByteFile`: `available() == 0`, always * `testSeekReadClosedFile` : call available() on an empty file, it should raise some IllegalStateException or IOE, but not an NPE For some of the other tests, I think you insert a couple of checks to say " available > 0" after a seek + read() Call. This also clarifies that "what should be available after a seek but before a read()?" As for cloudstores with lazy seek, available == 0, though if that breaks gzip then maybe they should return 1, so the assert should be available >1 with an option in the contract to say "actually returns 0". It's a tricky one to test on because really, the sole asserts should be 1. fails if the stream is closed (unless its a 0?) 1. returns >= if the stream is open 1. is always < remaining file length. Probably assert #3 is the one to check for: available() is valid when it is >=0 and <= (filelength -getPos()), with care taken about that second clause to make sure there's no off-by-one errors in the assert/code > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660748#comment-16660748 ] lqjacklee commented on HADOOP-15870: [~ste...@apache.org] Please review the code , and provide the suggestion about the test. https://github.com/apache/hadoop/pull/433 > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660747#comment-16660747 ] ASF GitHub Bot commented on HADOOP-15870: - GitHub user lqjack opened a pull request: https://github.com/apache/hadoop/pull/433 HADOOP-15870 Fix the nextPos You can merge this pull request into a Git repository by running: $ git pull https://github.com/lqjack/hadoop HADOOP-15870 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #433 commit ab1b1efb1aa5213c07db18ad29146908ac7c206d Author: lqjaclee Date: 2018-10-23T14:38:02Z HADOOP-15870 Fix the nextPos > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660549#comment-16660549 ] Steve Loughran commented on HADOOP-15870: - BTW, will be happy for a fix; I see just using getPos() should be enough too. We're pretty fussy about patch testing for the stores, but if you add contract tests I'll help broaden out the testing to cover the other stores. FWIW, clearly is is a rarely used API. Any specific format were you using here/library that breaks? > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.4, 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15870) S3AInputStream.remainingInFile should use nextReadPos
[ https://issues.apache.org/jira/browse/HADOOP-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660544#comment-16660544 ] Steve Loughran commented on HADOOP-15870: - ooh, you are right. And it surfaces into available(). Good find. This is something which we should add to the contract tests, as all FS implementations should be consistent here. > S3AInputStream.remainingInFile should use nextReadPos > - > > Key: HADOOP-15870 > URL: https://issues.apache.org/jira/browse/HADOOP-15870 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.1 >Reporter: Shixiong Zhu >Priority: Major > > Otherwise `remainingInFile` will not change after `seek`. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
