[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410403#comment-16410403 ] Hudson commented on HDFS-9643: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-9643. libhdfs++: Support async cancellation of read operations. (james.clampffer: rev 166b3d49df64afc6e42fd2dec6444d577bfb8f1b) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/continuation.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/cancel_tracker.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/datatransfer_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/remote_block_reader_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/datatransfer.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/bad_datanode_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/cancel_tracker.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/readergroup.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/readergroup.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-8643.HDFS-8707.004.patch, > HDFS-9643.HDFS-8707.000.patch, HDFS-9643.HDFS-8707.001.patch, > HDFS-9643.HDFS-8707.002.patch, HDFS-9643.HDFS-8707.003.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113275#comment-15113275 ] Bob Hansen commented on HDFS-9643: -- +1 > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8643.HDFS-8707.004.patch, > HDFS-9643.HDFS-8707.000.patch, HDFS-9643.HDFS-8707.001.patch, > HDFS-9643.HDFS-8707.002.patch, HDFS-9643.HDFS-8707.003.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112629#comment-15112629
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
25s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 39s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 42s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 39s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 34s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783839/HDFS-8643.HDFS-8707.004.patch
|
| JIRA Issue | HDFS-9643 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux ea6ad3c0caef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14205/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 76MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14205/console |
This message was automatically generated.
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
>
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110720#comment-15110720
]
Bob Hansen commented on HDFS-9643:
--
In {{ReaderGroup::GetLiveReaders()}}, you need to trim the readers_ member once
you're done with {{remove_if}}. {{remove_if}} only shuffles the live elements
forward and returns the new end; you need a subsequent {{readers_.erase(newEnd,
readers_.end());}}
What does the comment "no-op to keep method PV qualified in parent" mean?
I don't see an implementation of {{DataNodeConnection::Cancel()}}. Did I miss
something?
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-9643.HDFS-8707.000.patch,
> HDFS-9643.HDFS-8707.001.patch, HDFS-9643.HDFS-8707.002.patch
>
>
> It should be possible for any thread to cancel operations in progress on a
> FileHandle. Any ephemeral objects created by the FileHandle should free
> resources as quickly as possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110973#comment-15110973 ] James Clampffer commented on HDFS-9643: --- "you need to trim the readers_ member once you're done with remove_if. remove_if only shuffles the live elements forward" Good catch, I didn't expect it to work like that. "What does the comment "no-op to keep method PV qualified in parent" mean?" I was mixed up with cv-qualified being a real acronym; for some reason I thought pv was too when I wrote that. I wanted the parent's method to be declared pure virtual but that meant I had put stubs in the testing classes. "I don't see an implementation of DataNodeConnection::Cancel(). Did I miss something?" It looked like the convention in datanodeconnection.h was to leave really tiny methods in the header so I just left it there. I can move it out to the implementation if you prefer. > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch, > HDFS-9643.HDFS-8707.001.patch, HDFS-9643.HDFS-8707.002.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111342#comment-15111342
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
39s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 23m 4s {color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_66 with JDK
v1.8.0_66 generated 1 new + 2 unchanged - 1 fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 1s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 57s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 18s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783653/HDFS-9643.HDFS-8707.003.patch
|
| JIRA Issue | HDFS-9643 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux b21f0d1663d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| cc | hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_66:
https://builds.apache.org/job/PreCommit-HDFS-Build/14189/artifact/patchprocess/diff-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_66.txt
|
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14189/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111305#comment-15111305 ] Bob Hansen commented on HDFS-9643: -- Ah, I missed it in the header because I was looking for a close, not a cancel. According to the ASIO docs (http://www.boost.org/doc/libs/1_60_0/doc/html/boost_asio/reference/basic_stream_socket/cancel/overload2.html), connection::cancel() is really not reliable, and we are much better closing the connection rather than cancelling it. Other than that, +1. > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch, > HDFS-9643.HDFS-8707.001.patch, HDFS-9643.HDFS-8707.002.patch, > HDFS-9643.HDFS-8707.003.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109853#comment-15109853
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 8s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 14s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 55s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 53s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783465/HDFS-9643.HDFS-8707.002.patch
|
| JIRA Issue | HDFS-9643 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 140c72187fc8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14179/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 76MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14179/console |
This message was automatically generated.
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
>
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108785#comment-15108785 ] James Clampffer commented on HDFS-9643: --- Your observation is right; if network IO hangs or is just really slow this won't really help out. In order to close bad connections in a timely manner something would need to keep track of BlockReader instances and push out updates rather than having them poll the cancelled flag. I think this could be done by having the FileHandleImpl keep a vectorof readers it instantiates. When CancelOperations is called the FH could iterate through that vector and close the socket of any readers that are still alive. Downside would be that the vector would need some periodic garbage collection to avoid accumulating a lot of weak_ptrs to nothing. The idea of adding GC that loops over an operation that will be locking the data bus doesn't sound ideal so I'm open to suggestions. Otherwise I'll try and implement the method I described and post a patch later today. > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch, > HDFS-9643.HDFS-8707.001.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107627#comment-15107627
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
47s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 9s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 8s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 9s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 0s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783190/HDFS-9643.HDFS-8707.001.patch
|
| JIRA Issue | HDFS-9643 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux e0d6882aa5ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / b1fbec1 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_66
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 |
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/14161/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 79MB |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14161/console |
This message was automatically generated.
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
> Project:
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107693#comment-15107693 ] Bob Hansen commented on HDFS-9643: -- That is looking very good. When running on a TCP connection (rather than in unit tests), when a request is hanging because the server has gone catatonic (say... thrashing in GC), how does the current patch tickle the pipeline to check the cancel flag and bail out? I assumed that the only way to do that reliably was to close the tcp connection when the handle is cancelled (see http://www.boost.org/doc/libs/1_60_0/doc/html/boost_asio/reference/basic_stream_socket/cancel/overload2.html), and I don't see that in this patch. Is there something else going on that I have missed? > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch, > HDFS-9643.HDFS-8707.001.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102252#comment-15102252 ] Bob Hansen commented on HDFS-9643: -- Thanks for jumping into that [~James Clampffer]. It looks like a very good start. Rather than having a lock in the cancellation handle, we should be able to get away with a std::atomic. bq. -Right now the cancel logic is added directly to each continuation in the remote block reader. On one hand this is simple and works, on the other it's boilerplate code. Is this worth pushing into the continuation pipeline code at the moment? I think it's worth keeping it simple until NN operations become cancelable. The pipeline class already has a concept of annihilating error handling - if !status.ok(), skip over the rest of the pipeline and deliver an error. That would seem to be an opportune place to put in a cancellation check. -In this implementation FileHandle::CancelOperations is irreversible and prevents it from being used again. Can anyone think of a reason not to have it also close the file or at least clear vector? I think we need to have it close the file so that any outstanding async requests get their callback, and can respond to the cancellation. Having well-defined behavior after cancellation for stateful objects can be tricky; what should the position be? Are there cases where continuing streaming reads on a file with an undefined position would be useful? I suppose they could reset the position to a known good one and continue reading. Cancelling the file handle, but still able to efficiently do preads is a nice feature. Cancelling in-flight preads is another aspect to be handled (although that could be a different JIRA. -Should the FileHandle have a callback when it knows that there are no pending operations? Should be possible to just check the reference count on the CancelHandle to verify. It would need to be holding a lock preventing additional operations for that to make sense, and calling into consumer code while holding a lock is always a dangerous proposition. What's the use case for the callback? Is it compelling? > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[ https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102641#comment-15102641 ] James Clampffer commented on HDFS-9643: --- Thanks for checking this out [~bobhansen]! "Rather than having a lock in the cancellation handle, we should be able to get away with a std::atomic." Sounds good. I originally went down that path, there was some reason I switched to a lock that I can't remember at the moment. It certainly looks like I can switch back to std::atomic_bool. "The pipeline class already has a concept of annihilating error handling - if !status.ok(), skip over the rest of the pipeline and deliver an error. That would seem to be an opportune place to put in a cancellation check." Sounds good to me. "I think we need to have it close the file so that any outstanding async requests get their callback, and can respond to the cancellation. Having well-defined behavior after cancellation for stateful objects can be tricky; what should the position be? Are there cases where continuing streaming reads on a file with an undefined position would be useful? I suppose they could reset the position to a known good one and continue reading." I agree. For now maybe it's worth keeping it simple by avoiding doing much of anything other than setting the flag when CancelOperations is called on the FH and preventing that FH from initiating new operations. It needs to stay around long enough for the ephemeral objects to see that the flag has changed. "Cancelling the file handle, but still able to efficiently do preads is a nice feature. Cancelling in-flight preads is another aspect to be handled (although that could be a different JIRA." In flight in the sense that control has been handed over to asio? I'd like to push that into another jira for now to keep this patch relatively simple but I think that's very important to have working as well. "It would need to be holding a lock preventing additional operations for that to make sense, and calling into consumer code while holding a lock is always a dangerous proposition. What's the use case for the callback? Is it compelling?" The case I had in mind was having a synchronous application that does a bunch of prep work to set up it's own async ephemeral objects that in turn call into libhdfs++. If that application wants to tear down it's ephemeral object(s) and reclaim buffers as quickly as possible it'd be nice to be able to check that it's safe to do so without each operation callback notifying the synchronous bits that the work is completed. The more I think this case out the more it seems like that sort application should just be using the synchronous interfaces. > libhdfs++: Support async cancellation of read operations > > > Key: HDFS-9643 > URL: https://issues.apache.org/jira/browse/HDFS-9643 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9643.HDFS-8707.000.patch > > > It should be possible for any thread to cancel operations in progress on a > FileHandle. Any ephemeral objects created by the FileHandle should free > resources as quickly as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9643) libhdfs++: Support async cancellation of read operations
[
https://issues.apache.org/jira/browse/HDFS-9643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096969#comment-15096969
]
Hadoop QA commented on HDFS-9643:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m 35s
{color} | {color:red} Docker failed to build yetus/hadoop:0cf5e66. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782121/HDFS-9643.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-9643 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14115/console |
This message was automatically generated.
> libhdfs++: Support async cancellation of read operations
>
>
> Key: HDFS-9643
> URL: https://issues.apache.org/jira/browse/HDFS-9643
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-9643.HDFS-8707.000.patch
>
>
> It should be possible for any thread to cancel operations in progress on a
> FileHandle. Any ephemeral objects created by the FileHandle should free
> resources as quickly as possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
