[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427442#comment-16427442
]
genericqa commented on HDFS-13400:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13400 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917626/HDFS-13400-unit-test.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 04486598d732 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e52539b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23796/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23796/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427233#comment-16427233
]
Erik Krogen commented on HDFS-13400:
Yup, it's just the {{getPos()}} that messes up. Data is still correctly
appended to the end of the file.
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427231#comment-16427231
]
Íñigo Goiri commented on HDFS-13400:
The {{write()}} works correctly and appends to the end, right?
Making sure it's just an issue with {{getPos()}}.
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427218#comment-16427218
]
Erik Krogen commented on HDFS-13400:
Sure. Sorry for fighting with you on JIRA status, just didn't want to make it
seem like someone had posted a patch fixing the issue.
So the issue comes from here:
{code}
/**
* Handle create/append output streams
*/
class FsPathOutputStreamRunner
extends AbstractFsPathRunner {
private final int bufferSize;
FsPathOutputStreamRunner(Op op, Path fspath, int bufferSize,
Param... parameters) {
super(op, fspath, parameters);
this.bufferSize = bufferSize;
}
@Override
FSDataOutputStream getResponse(final HttpURLConnection conn)
throws IOException {
return new FSDataOutputStream(new BufferedOutputStream(
conn.getOutputStream(), bufferSize), statistics) {
@Override
public void close() throws IOException {
try {
super.close();
} finally {
try {
validateResponse(op, conn, true);
} finally {
// This is a connection to DataNode. Let's disconnect since
// there is little chance that the connection will be reused
// any time soonl
conn.disconnect();
}
}
}
};
}
}
{code}
We see that the {{FSDataOutputStream}} is being created using the two-arg
constructor, but we need to use the three-arg constructor which allows for
specifying a nonzero {{startPosition}}. That part is easy. How to get the
correct {{startPosition}} is hard...
The attached test simply fails on the assert:
{code}
java.lang.AssertionError:
Expected :10
Actual :0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.web.TestWebHDFS.testWebHdfsSuccessfulAppend(TestWebHDFS.java:1498)
{code}
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427210#comment-16427210
]
Íñigo Goiri commented on HDFS-13400:
[~xkrogen], I wanted to get the exception catch by Yetus for tracking.
Do you mind posting the log/failed condition?
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427205#comment-16427205
]
Erik Krogen commented on HDFS-13400:
[~elgoiri] FYI cancelling Patch Available because what I have posted does not
contain any sort of fix...
This ends up being kind of tricky. The position of the stream is not known
until the DataNode is contacted and it performs {{DFSClient#append}}. In the
current setup, the DataNode does not send a response until it is done writing
the file... So I don't see a clean way for it to communicate an initial
position back to the client.
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13400) WebHDFS append returned stream has incorrectly set position
[
https://issues.apache.org/jira/browse/HDFS-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426346#comment-16426346
]
Erik Krogen commented on HDFS-13400:
Attaching a patch with a unit test which demonstrates the issue
> WebHDFS append returned stream has incorrectly set position
> ---
>
> Key: HDFS-13400
> URL: https://issues.apache.org/jira/browse/HDFS-13400
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.1
>Reporter: Erik Krogen
>Priority: Minor
> Attachments: HDFS-13400-unit-test.patch
>
>
> If you call {{getPos()}} on an {{FSDataOutputStream}} returned by
> {{WebHdfsFileSystem#append}}, it will initially return 0, even if the file
> already had content. This method should return the initial length of the file
> before appending.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
