[jira] [Updated] (HDFS-11201) Spelling errors in the logging, help, assertions and exception messages
[ https://issues.apache.org/jira/browse/HDFS-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HDFS-11201: -- Fix Version/s: 3.0.0-alpha2 Status: Patch Available (was: Open) > Spelling errors in the logging, help, assertions and exception messages > --- > > Key: HDFS-11201 > URL: https://issues.apache.org/jira/browse/HDFS-11201 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, diskbalancer, httpfs, namenode, nfs >Affects Versions: 3.0.0-alpha1 >Reporter: Grant Sohn >Priority: Trivial > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11201.1.patch > > > Found a set of spelling errors in the user-facing code. > Examples are: > odlest -> oldest > Illagal -> Illegal > bounday -> boundary -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11201) Spelling errors in the logging, help, assertions and exception messages
[ https://issues.apache.org/jira/browse/HDFS-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Sohn updated HDFS-11201: -- Attachment: HDFS-11201.1.patch Fixes for the various spelling errors. > Spelling errors in the logging, help, assertions and exception messages > --- > > Key: HDFS-11201 > URL: https://issues.apache.org/jira/browse/HDFS-11201 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, diskbalancer, httpfs, namenode, nfs >Affects Versions: 3.0.0-alpha1 >Reporter: Grant Sohn >Priority: Trivial > Attachments: HDFS-11201.1.patch > > > Found a set of spelling errors in the user-facing code. > Examples are: > odlest -> oldest > Illagal -> Illegal > bounday -> boundary -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11201) Spelling errors in the logging, help, assertions and exception messages
Grant Sohn created HDFS-11201: - Summary: Spelling errors in the logging, help, assertions and exception messages Key: HDFS-11201 URL: https://issues.apache.org/jira/browse/HDFS-11201 Project: Hadoop HDFS Issue Type: Bug Components: datanode, diskbalancer, httpfs, namenode, nfs Affects Versions: 3.0.0-alpha1 Reporter: Grant Sohn Priority: Trivial Found a set of spelling errors in the user-facing code. Examples are: odlest -> oldest Illagal -> Illegal bounday -> boundary -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[
https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15719042#comment-15719042
]
Hadoop QA commented on HDFS-11197:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.viewfs.TestViewFsAtHdfsRoot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841653/HDFS-11197-6.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cdb5be01dea0 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f885160 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17752/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17752/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17752/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17752/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Listing encryption zones fails when deleting a EZ that is on a snapshotted
> directory
>
>
> Key: HDFS-11197
>
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[
https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-11197:
Status: Patch Available (was: In Progress)
> Listing encryption zones fails when deleting a EZ that is on a snapshotted
> directory
>
>
> Key: HDFS-11197
> URL: https://issues.apache.org/jira/browse/HDFS-11197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch,
> HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch
>
>
> If a EZ directory is under a snapshotable directory, and a snapshot has been
> taking, then if this EZ is permanently deleted, it causes *hdfs crypto
> listZones* command to fail without showing any of the still available zones.
> This happens only after the EZ is removed from Trash folder. For example,
> considering */test-snap* folder is snapshotable and there is already an
> snapshot for it:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /test-snap/EZ-1
> INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at:
> hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> /user/systest my-key
> /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1
> Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> RemoteException: Absolute path required
> {noformat}
> Once this error happens, *hdfs crypto -listZones* only works again if we
> remove the snapshot:
> {noformat}
> $ hdfs dfs -deleteSnapshot /test-snap snap1
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> If we instead delete the EZ using *skipTrash* option, *hdfs crypto
> -listZones* does not break:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-2 my-key
> $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2
> Deleted /test-snap/EZ-2
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> The different behaviour seems to be because when removing the EZ trash
> folder, it's related INode is left with no parent INode. This causes
> *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when
> trying to resolve the inodes in the given path.
> Am proposing a patch that fixes this issue by simply performing an additional
> check on *EncryptionZoneManager.listEncryptionZones* for the case an inode
> has no parent, so that it would be skipped on the list without trying to
> resolve it. Feedback on the proposal is appreciated.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[
https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-11197:
Attachment: HDFS-11197-6.patch
Thanks again, [~xiaochen]! Sending another patch attempt with your last
suggestions.
Apologies for the white space previously. I actually tried to run the {{git
apply --whitespace=fix <>}} on that patch to fix white spaces, but
I guess it didn't work (or maybe I'm doing something wrong). Anyway, I believe
it has no white spaces now. Let me know if you find any additional enhancements.
> Listing encryption zones fails when deleting a EZ that is on a snapshotted
> directory
>
>
> Key: HDFS-11197
> URL: https://issues.apache.org/jira/browse/HDFS-11197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch,
> HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch
>
>
> If a EZ directory is under a snapshotable directory, and a snapshot has been
> taking, then if this EZ is permanently deleted, it causes *hdfs crypto
> listZones* command to fail without showing any of the still available zones.
> This happens only after the EZ is removed from Trash folder. For example,
> considering */test-snap* folder is snapshotable and there is already an
> snapshot for it:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /test-snap/EZ-1
> INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at:
> hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> /user/systest my-key
> /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1
> Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> RemoteException: Absolute path required
> {noformat}
> Once this error happens, *hdfs crypto -listZones* only works again if we
> remove the snapshot:
> {noformat}
> $ hdfs dfs -deleteSnapshot /test-snap snap1
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> If we instead delete the EZ using *skipTrash* option, *hdfs crypto
> -listZones* does not break:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-2 my-key
> $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2
> Deleted /test-snap/EZ-2
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> The different behaviour seems to be because when removing the EZ trash
> folder, it's related INode is left with no parent INode. This causes
> *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when
> trying to resolve the inodes in the given path.
> Am proposing a patch that fixes this issue by simply performing an additional
> check on *EncryptionZoneManager.listEncryptionZones* for the case an inode
> has no parent, so that it would be skipped on the list without trying to
> resolve it. Feedback on the proposal is appreciated.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11197) Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
[
https://issues.apache.org/jira/browse/HDFS-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HDFS-11197:
Status: In Progress (was: Patch Available)
> Listing encryption zones fails when deleting a EZ that is on a snapshotted
> directory
>
>
> Key: HDFS-11197
> URL: https://issues.apache.org/jira/browse/HDFS-11197
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-11197-1.patch, HDFS-11197-2.patch,
> HDFS-11197-3.patch, HDFS-11197-4.patch, HDFS-11197-5.patch, HDFS-11197-6.patch
>
>
> If a EZ directory is under a snapshotable directory, and a snapshot has been
> taking, then if this EZ is permanently deleted, it causes *hdfs crypto
> listZones* command to fail without showing any of the still available zones.
> This happens only after the EZ is removed from Trash folder. For example,
> considering */test-snap* folder is snapshotable and there is already an
> snapshot for it:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /test-snap/EZ-1
> INFO fs.TrashPolicyDefault: Moved: 'hdfs://ns1/test-snap/EZ-1' to trash at:
> hdfs://ns1/user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> /user/systest my-key
> /user/hdfs/.Trash/Current/test-snap/EZ-1 my-key
> $ hdfs dfs -rmr /user/hdfs/.Trash/Current/test-snap/EZ-1
> Deleted /user/hdfs/.Trash/Current/test-snap/EZ-1
> $ hdfs crypto -listZones
> RemoteException: Absolute path required
> {noformat}
> Once this error happens, *hdfs crypto -listZones* only works again if we
> remove the snapshot:
> {noformat}
> $ hdfs dfs -deleteSnapshot /test-snap snap1
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> If we instead delete the EZ using *skipTrash* option, *hdfs crypto
> -listZones* does not break:
> {noformat}
> $ hdfs crypto -listZones
> /user/systest my-key
> /test-snap/EZ-2 my-key
> $ hdfs dfs -rmr -skipTrash /test-snap/EZ-2
> Deleted /test-snap/EZ-2
> $ hdfs crypto -listZones
> /user/systest my-key
> {noformat}
> The different behaviour seems to be because when removing the EZ trash
> folder, it's related INode is left with no parent INode. This causes
> *EncryptionZoneManager.listEncryptionZones* to throw the seen error, when
> trying to resolve the inodes in the given path.
> Am proposing a patch that fixes this issue by simply performing an additional
> check on *EncryptionZoneManager.listEncryptionZones* for the case an inode
> has no parent, so that it would be skipped on the list without trying to
> resolve it. Feedback on the proposal is appreciated.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11189) WebHDFS and FollowRedirects
[ https://issues.apache.org/jira/browse/HDFS-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718649#comment-15718649 ] Oscar Guadilla commented on HDFS-11189: --- Hi Weiwei Yang, I will try to explain it better. Let's focus on a GET operation: - step 1 --> http request to the namenode from the client to get a concrete file - step 2 --> the namenode looks up where is the file and sends to the client a 307 (forward) response with the datanode uri in the location - step 3 --> the client reads the response (307) and makes a second http client to the suggested datanode - step 4 --> the datanode reads the requested file and gives it back the content to the client It works fine if the httpconnection has the default value (true) in the getFollowRedirects atributte; otherwise if fails because it stops in the step3. I know that this atributte followRedirects should be by default true but we detected that certain libraries modify that behaviour and webhdfs worked poorly. To solve this problem what we did was change the FileSystem class to instead of supposing that FollowRedirects is true always set the value to true. Of course we used the instance scope using setInstanceFollowRedirects instead of using the global parameter. PS: It was a nighmare of two weeks for us and we wanted to contribute with the solution we found. We are just users of hadoop and we wanted to give back something from us to the project. Regards, Oscar > WebHDFS and FollowRedirects > --- > > Key: HDFS-11189 > URL: https://issues.apache.org/jira/browse/HDFS-11189 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.3 > Environment: We are using webhdfs from a j2ee environment with java > 6. In the web application we have a lot of libraries: spring, jersey, > jackson... >Reporter: Oscar Guadilla > > In some cases with simple operations (get file - non two step operation) we > detect that FollowRedirects flag of http comes as "false" and it makes > webhdfs crash. > We don't really know which library change this behavior and it is really > strange because the change is done directly in the jvm because in other > wars/ears of the same j2ee server it fails also. If we restart the j2ee > server it starts working again. > To fix the problem we changed the WebHdfsFileSystem class adding > "setInstanceFollowRedirects(true)" in the connection management instead of > supposing that it should be true and it works fine. > The same problem arises both in 1.x and 2.x webhdfs. We didn't test in 3.x > Could you fix it? We did it in our environment but it would fine in it could > be included in the next releases. > Thanks in advance, > Oscar -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8630) WebHDFS : Support get/set/unset StoragePolicy
[
https://issues.apache.org/jira/browse/HDFS-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718570#comment-15718570
]
Hadoop QA commented on HDFS-8630:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 37 new
+ 666 unchanged - 6 fixed = 703 total (was 672) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m
7s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
22s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8630 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841634/HDFS-8630.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ce8dd0f76de3 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f885160 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17751/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17751/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
[jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance
[
https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718536#comment-15718536
]
Ming Ma commented on HDFS-10206:
ok. Maybe it isn't precise way to refer it. The "network path" comes from
NodeBase#getPath method. Anyway, the point is the new method should return 0 in
case of two identical nodes.
> getBlockLocations might not sort datanodes properly by distance
> ---
>
> Key: HDFS-10206
> URL: https://issues.apache.org/jira/browse/HDFS-10206
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Nandakumar
> Attachments: HDFS-10206.000.patch, HDFS-10206.001.patch,
> HDFS-10206.002.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some
> datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}}
> might not put the local-rack datanodes at the beginning of the sorted list.
> That is because the function didn't call {{networktopology.add(client);}} to
> properly set the node's parent node; something required by
> {{networktopology.sortByDistance}} to compute distance between two nodes in
> the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only
> distinguishes local rack from remote rack, but it doesn't support general
> distance calculation to tell how remote the rack is.
> {noformat}
> NetworkTopology.java
> protected int getWeight(Node reader, Node node) {
> // 0 is local, 1 is same rack, 2 is off rack
> // Start off by initializing to off rack
> int weight = 2;
> if (reader != null) {
> if (reader.equals(node)) {
> weight = 0;
> } else if (isOnSameRack(reader, node)) {
> weight = 1;
> }
> }
> return weight;
> }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to
> address another issue. Regardless of where we do the sorting, we still need
> fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object
> used by DatanodeManager and requires Nodes stored in the topology to be
> {{DatanodeDescriptor}} for block placement. So we need to make sure we don't
> pollute the NetworkTopology if we plan to fix it on the server side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11200) build hadoop in windows 7/10 64 bit using Visual C++ Build Tools Standalone compiler, dotnet 4.5
[ https://issues.apache.org/jira/browse/HDFS-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved HDFS-11200. - Resolution: Fixed your system installed with Visual Studio 2010..? If yes,then you might need to install dotnet4.0..Check the following link for details. http://stackoverflow.com/questions/12390175/targeting-net-framework-4-5-via-visual-studio-2010. As of now ,I am going to close this as *not a problem* . If you have any queries,please post in [user mailing list|http://hadoop.apache.org/mailing_lists.html],Jira is track the issues. > build hadoop in windows 7/10 64 bit using Visual C++ Build Tools Standalone > compiler, dotnet 4.5 > > > Key: HDFS-11200 > URL: https://issues.apache.org/jira/browse/HDFS-11200 > Project: Hadoop HDFS > Issue Type: Wish > Components: build >Affects Versions: 2.7.4 > Environment: windows 7/10 64 bit, Visual C++ Build Tools Standalone > compiler, libraries. >Reporter: Mmohamed Hassan > > configure maven , cmake to build the new version hadoop 2.7.4 to be packaged > using Visual C++ Build Tools Standalone compiler, libraries, dotnet 4.5 > (http://landinghub.visualstudio.com/visual-cpp-build-tools) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11199) failure to build hadoop 2.7.1 in windows 7 64 bit
[ https://issues.apache.org/jira/browse/HDFS-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved HDFS-11199. - Resolution: Not A Problem Tags: (was: hadoop v2.7.1 hdfs) Looks to be cmake is not installed.Hope following link can help you for windows compilation. http://zutai.blogspot.com/2014/06/build-install-and-run-hadoop-24-240-on.html?showComment=1422091525887#c2264594416650430988. As of now ,I am going to close this *not a problem* . If you have any queries,please post in [user mailing list|http://hadoop.apache.org/mailing_lists.html],Jira is track the issues. > failure to build hadoop 2.7.1 in windows 7 64 bit > - > > Key: HDFS-11199 > URL: https://issues.apache.org/jira/browse/HDFS-11199 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 > Environment: windows 7 64 bit using windows sdk 7.1, the setting of > command line is: > APPVER=6.1 > CL=/AI C:\Windows\Microsoft.NET\Framework64\v4.0.30319 > CommandPromptType=Native > CURRENT_CPU=x64 > FrameworkVersion=v4.0.30319 > platform=x64 > PlatformToolset=Windows7.1SDK > PROCESSOR_ARCHITECTURE=AMD64 > sdkdir=C:\Program Files\Microsoft SDKs\Windows\v7.1\ > TARGET_CPU=x64 > TARGET_PLATFORM=WIN7 > ToolsVersion=4.0 > VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio > 10.0\Common7\Tools\ > WindowsSDKDir=C:\Program Files\Microsoft SDKs\Windows\v7.1\ > WindowsSDKVersionOverride=v7.1 >Reporter: Mmohamed Hassan > > In windows 7 64 bit, I build hadoop version 2.7.1 > i installed all needed software, > for c compiler i use the c++ compiler of windows sdk 7.1 (visual studio 2010 > isn't installed) > I run from Windows SDK 7.1 Command Prompt with release x64 > but the build failed with errors > The C compiler identification is unknown > -- The CXX compiler identification is unknown > CMake Error in CMakeLists.txt: >No CMAKE_C_COMPILER could be found. > > CMake Error in CMakeLists.txt: > No CMAKE_CXX_COMPILER could be found. > The following is the console output > [INFO] > > [INFO] Building Apache Hadoop HDFS 2.7.1 > [INFO] > > [INFO] > [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-os) @ hadoop-hdfs --- > > > main: > [INFO] Executed tasks > [INFO] > [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-hdfs --- > [INFO] Executing tasks > main: > [exec] -- The C compiler identification is unknown > [exec] -- The CXX compiler identification is unknown > [exec] CMake Error in CMakeLists.txt: > [exec] No CMAKE_C_COMPILER could be found. > [exec] > [exec] > [exec] > [exec] CMake Error in CMakeLists.txt: > [exec] No CMAKE_CXX_COMPILER could be found. > [exec] > [exec] > [exec] > [exec] -- Configuring incomplete, errors occurred! > [exec] See also > "E:/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs/target > /native/CMakeFiles/CMakeOutput.log". > [exec] See also > "E:/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs/target > /native/CMakeFiles/CMakeError.log". > [INFO] > > [INFO] Reactor Summary: > [INFO] > [INFO] Apache Hadoop Main . SUCCESS [ 2.995 > s] > [INFO] Apache Hadoop Project POM .. SUCCESS [ 4.477 > s] > [INFO] Apache Hadoop Annotations .. SUCCESS [ 4.696 > s] > [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.250 > s] > [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 3.759 > s] > [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.775 > s] > [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 3.354 > s] > [INFO] Apache Hadoop Auth . SUCCESS [ 4.056 > s] > [INFO] Apache Hadoop Auth Examples SUCCESS [ 3.807 > s] > [INFO] Apache Hadoop Common ... SUCCESS [02:09 > min] > [INFO] Apache Hadoop NFS .. SUCCESS [ 12.776 > s] > [INFO] Apache Hadoop KMS .. SUCCESS [ 15.304 > s] > [INFO] Apache Hadoop Common Project ... SUCCESS [ 0.031 > s] > [INFO] Apache Hadoop HDFS . FAILURE [ 42.105 > s] > [INFO] Apache Hadoop HttpFS ... SKIPPED > [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED > . > . > [INFO] > > [INFO] BUILD FAILURE >
[jira] [Updated] (HDFS-8630) WebHDFS : Support get/set/unset StoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-8630: - Attachment: HDFS-8630.010.patch > WebHDFS : Support get/set/unset StoragePolicy > -- > > Key: HDFS-8630 > URL: https://issues.apache.org/jira/browse/HDFS-8630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Reporter: nijel >Assignee: Surendra Singh Lilhore > Attachments: HDFS-8630.001.patch, HDFS-8630.002.patch, > HDFS-8630.003.patch, HDFS-8630.004.patch, HDFS-8630.005.patch, > HDFS-8630.006.patch, HDFS-8630.007.patch, HDFS-8630.008.patch, > HDFS-8630.009.patch, HDFS-8630.010.patch, HDFS-8630.patch > > > User can set and get the storage policy from filesystem object. Same > operation can be allowed trough REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8630) WebHDFS : Support get/set/unset StoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-8630: - Attachment: (was: HDFS-8630.010.patch) > WebHDFS : Support get/set/unset StoragePolicy > -- > > Key: HDFS-8630 > URL: https://issues.apache.org/jira/browse/HDFS-8630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Reporter: nijel >Assignee: Surendra Singh Lilhore > Attachments: HDFS-8630.001.patch, HDFS-8630.002.patch, > HDFS-8630.003.patch, HDFS-8630.004.patch, HDFS-8630.005.patch, > HDFS-8630.006.patch, HDFS-8630.007.patch, HDFS-8630.008.patch, > HDFS-8630.009.patch, HDFS-8630.patch > > > User can set and get the storage policy from filesystem object. Same > operation can be allowed trough REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718227#comment-15718227
]
Manjunath edited comment on HDFS-11182 at 12/3/16 3:17 PM:
---
[~arpitagarwal] I am little confused about the reason for the change in
DatasetVolumeChecker.checkAllVolume methods to use Semaphore and CoutDownLatch
instead of the earlier approach of using CountDownLatch.
The confusion is whether the CountDownLatch countdown is correctly called after
all threads scheduled for completing the volume check finish their jobs. I am
presenting my understanding as to why the CountDownLatch countdown is called by
the first thread completing the volume check which would result in incorrect
expected behaviour. Please correct me if I am wrong.
The below code acquires all references.size()-1 permits .
{code}
final Semaphore semaphore = new Semaphore(-references.size() + 1);
{code}
The invokeCallback releases permit in Semaphore and then does a tryAcquire
which would allow it to aquire the permit it just released and hence would go
and call the callback.call method which would release the latch by the
countDown.
So it looks like the first thread which completed the volume check will release
the latch which would mean we havent waited for maxAllowedTimeForCheckMs by the
await call made on the latch.
{code}
private void invokeCallback() {
try {
semaphore.release();
if (callback != null && semaphore.tryAcquire()) {
callback.call(healthyVolumes, failedVolumes);
}
} catch(Exception e) {
// Propagating this exception is unlikely to be helpful.
LOG.warn("Unexpected exception", e);
}
}
Futures.addCallback(future, new ResultHandler(
- reference, healthyVolumes, failedVolumes, resultsLatch, null));
+ reference, healthyVolumes, failedVolumes, semaphore, new
Callback() {
+@Override
+public void call(Set ignored1,
+ Set ignored2) {
+ latch.countDown();
+}
+ }));
{code}
Please suggest on this.
was (Author: manju_hadoop):
[~arpitagarwal] I am little confused about the reason for the change in
DatasetVolumeChecker.checkAllVolume methods to use Semaphore and CoutDownLatch
instead of the earlier approach of using CountDownLatch.
The confusion is whether the CountDownLatch countdown is correctly called after
all threads scheduled for completing the volume check finish their jobs.
The below code acquires all references.size()-1 permits .
{code}
final Semaphore semaphore = new Semaphore(-references.size() + 1);
{code}
The invokeCallback releases permit in Semaphore and then does a tryAcquire
which would allow it to aquire the permit it just released and hence would go
and call the callback.call method which would release the latch by the
countDown.
So it looks like the first thread which completed the volume check will release
the latch which would mean we havent waited for maxAllowedTimeForCheckMs by the
await call made on the latch.
{code}
private void invokeCallback() {
try {
semaphore.release();
if (callback != null && semaphore.tryAcquire()) {
callback.call(healthyVolumes, failedVolumes);
}
} catch(Exception e) {
// Propagating this exception is unlikely to be helpful.
LOG.warn("Unexpected exception", e);
}
}
Futures.addCallback(future, new ResultHandler(
- reference, healthyVolumes, failedVolumes, resultsLatch, null));
+ reference, healthyVolumes, failedVolumes, semaphore, new
Callback() {
+@Override
+public void call(Set ignored1,
+ Set ignored2) {
+ latch.countDown();
+}
+ }));
{code}
Please suggest on this.
> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in
> HDFS-11149 to parallelize disk checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718227#comment-15718227
]
Manjunath commented on HDFS-11182:
--
[~arpitagarwal] I am little confused about the reason for the change in
DatasetVolumeChecker.checkAllVolume methods to use Semaphore and CoutDownLatch
instead of the earlier approach of using CountDownLatch.
The confusion is whether the CountDownLatch countdown is correctly called after
all threads scheduled for completing the volume check finish their jobs.
The below code acquires all references.size()-1 permits .
{code}
final Semaphore semaphore = new Semaphore(-references.size() + 1);
{code}
The invokeCallback releases permit in Semaphore and then does a tryAcquire
which would allow it to aquire the permit it just released and hence would go
and call the callback.call method which would release the latch by the
countDown.
So it looks like the first thread which completed the volume check will release
the latch which would mean we havent waited for maxAllowedTimeForCheckMs by the
await call made on the latch.
{code}
private void invokeCallback() {
try {
semaphore.release();
if (callback != null && semaphore.tryAcquire()) {
callback.call(healthyVolumes, failedVolumes);
}
} catch(Exception e) {
// Propagating this exception is unlikely to be helpful.
LOG.warn("Unexpected exception", e);
}
}
Futures.addCallback(future, new ResultHandler(
- reference, healthyVolumes, failedVolumes, resultsLatch, null));
+ reference, healthyVolumes, failedVolumes, semaphore, new
Callback() {
+@Override
+public void call(Set ignored1,
+ Set ignored2) {
+ latch.countDown();
+}
+ }));
{code}
Please suggest on this.
> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in
> HDFS-11149 to parallelize disk checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11200) build hadoop in windows 7/10 64 bit using Visual C++ Build Tools Standalone compiler, dotnet 4.5
Mmohamed Hassan created HDFS-11200: -- Summary: build hadoop in windows 7/10 64 bit using Visual C++ Build Tools Standalone compiler, dotnet 4.5 Key: HDFS-11200 URL: https://issues.apache.org/jira/browse/HDFS-11200 Project: Hadoop HDFS Issue Type: Wish Components: build Affects Versions: 2.7.4 Environment: windows 7/10 64 bit, Visual C++ Build Tools Standalone compiler, libraries. Reporter: Mmohamed Hassan configure maven , cmake to build the new version hadoop 2.7.4 to be packaged using Visual C++ Build Tools Standalone compiler, libraries, dotnet 4.5 (http://landinghub.visualstudio.com/visual-cpp-build-tools) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11199) failure to build hadoop 2.7.1 in windows 7 64 bit
Mmohamed Hassan created HDFS-11199: -- Summary: failure to build hadoop 2.7.1 in windows 7 64 bit Key: HDFS-11199 URL: https://issues.apache.org/jira/browse/HDFS-11199 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 2.7.1 Environment: windows 7 64 bit using windows sdk 7.1, the setting of command line is: APPVER=6.1 CL=/AI C:\Windows\Microsoft.NET\Framework64\v4.0.30319 CommandPromptType=Native CURRENT_CPU=x64 FrameworkVersion=v4.0.30319 platform=x64 PlatformToolset=Windows7.1SDK PROCESSOR_ARCHITECTURE=AMD64 sdkdir=C:\Program Files\Microsoft SDKs\Windows\v7.1\ TARGET_CPU=x64 TARGET_PLATFORM=WIN7 ToolsVersion=4.0 VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\ WindowsSDKDir=C:\Program Files\Microsoft SDKs\Windows\v7.1\ WindowsSDKVersionOverride=v7.1 Reporter: Mmohamed Hassan In windows 7 64 bit, I build hadoop version 2.7.1 i installed all needed software, for c compiler i use the c++ compiler of windows sdk 7.1 (visual studio 2010 isn't installed) I run from Windows SDK 7.1 Command Prompt with release x64 but the build failed with errors The C compiler identification is unknown -- The CXX compiler identification is unknown CMake Error in CMakeLists.txt: No CMAKE_C_COMPILER could be found. CMake Error in CMakeLists.txt: No CMAKE_CXX_COMPILER could be found. The following is the console output [INFO] [INFO] Building Apache Hadoop HDFS 2.7.1 [INFO] [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-os) @ hadoop-hdfs --- main: [INFO] Executed tasks [INFO] [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-hdfs --- [INFO] Executing tasks main: [exec] -- The C compiler identification is unknown [exec] -- The CXX compiler identification is unknown [exec] CMake Error in CMakeLists.txt: [exec] No CMAKE_C_COMPILER could be found. [exec] [exec] [exec] [exec] CMake Error in CMakeLists.txt: [exec] No CMAKE_CXX_COMPILER could be found. [exec] [exec] [exec] [exec] -- Configuring incomplete, errors occurred! [exec] See also "E:/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs/target /native/CMakeFiles/CMakeOutput.log". [exec] See also "E:/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs/target /native/CMakeFiles/CMakeError.log". [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main . SUCCESS [ 2.995 s] [INFO] Apache Hadoop Project POM .. SUCCESS [ 4.477 s] [INFO] Apache Hadoop Annotations .. SUCCESS [ 4.696 s] [INFO] Apache Hadoop Assemblies ... SUCCESS [ 0.250 s] [INFO] Apache Hadoop Project Dist POM . SUCCESS [ 3.759 s] [INFO] Apache Hadoop Maven Plugins SUCCESS [ 3.775 s] [INFO] Apache Hadoop MiniKDC .. SUCCESS [ 3.354 s] [INFO] Apache Hadoop Auth . SUCCESS [ 4.056 s] [INFO] Apache Hadoop Auth Examples SUCCESS [ 3.807 s] [INFO] Apache Hadoop Common ... SUCCESS [02:09 min] [INFO] Apache Hadoop NFS .. SUCCESS [ 12.776 s] [INFO] Apache Hadoop KMS .. SUCCESS [ 15.304 s] [INFO] Apache Hadoop Common Project ... SUCCESS [ 0.031 s] [INFO] Apache Hadoop HDFS . FAILURE [ 42.105 s] [INFO] Apache Hadoop HttpFS ... SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED . . [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 03:55 min [INFO] Finished at: 2016-12-03T14:30:39+02:00 [INFO] Final Memory: 83M/494M [INFO] . -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8630) WebHDFS : Support get/set/unset StoragePolicy
[
https://issues.apache.org/jira/browse/HDFS-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718090#comment-15718090
]
Hadoop QA commented on HDFS-8630:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
44s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project: The patch generated 37 new
+ 666 unchanged - 6 fixed = 703 total (was 672) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
38s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 7 new + 8
unchanged - 0 fixed = 15 total (was 8) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 22s{color}
| {color:red} hadoop-hdfs-httpfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem |
| | hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-8630 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841628/HDFS-8630.010.patch |
| Optional
[jira] [Updated] (HDFS-8630) WebHDFS : Support get/set/unset StoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-8630: - Attachment: HDFS-8630.010.patch Thanks [~andrew.wang] for review.. v10 > bq. Although this is not related, could you add param documentation for "startAfter" as well? I added doc for "startafter" param. Please review.. > WebHDFS : Support get/set/unset StoragePolicy > -- > > Key: HDFS-8630 > URL: https://issues.apache.org/jira/browse/HDFS-8630 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Reporter: nijel >Assignee: Surendra Singh Lilhore > Attachments: HDFS-8630.001.patch, HDFS-8630.002.patch, > HDFS-8630.003.patch, HDFS-8630.004.patch, HDFS-8630.005.patch, > HDFS-8630.006.patch, HDFS-8630.007.patch, HDFS-8630.008.patch, > HDFS-8630.009.patch, HDFS-8630.010.patch, HDFS-8630.patch > > > User can set and get the storage policy from filesystem object. Same > operation can be allowed trough REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath edited comment on HDFS-11182 at 12/3/16 11:03 AM:
[~arpitagarwal] , sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync does the same
tasks except for few additional things in checkAllVolumes. The code can be
modified such that checkAllVolumes inturn calls the checkAllVolumesAsync and
save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
{code}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:-
{code} LOG.info("Scheduled health check for volume {}", reference.getVolume());
{code} this is there only in checkAllVolumes where as {code} LOG.info("Checking
{} volumes", references.size());{code} is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
was (Author: manju_hadoop):
[~arpitagarwal] , sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync does the same
tasks except for few additional things in checkAllVolumes. The code can be
modified such that checkAllVolumes inturn calls the checkAllVolumesAsync and
save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath edited comment on HDFS-11182 at 12/3/16 11:02 AM:
[~arpitagarwal] , sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync does the same
tasks except for few additional things in checkAllVolumes. The code can be
modified such that checkAllVolumes inturn calls the checkAllVolumesAsync and
save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
{code}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:- LOG.info("Scheduled health check for
volume {}", reference.getVolume()); this is there only in checkAllVolumes where
as LOG.info("Checking {} volumes", references.size()); is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
was (Author: manju_hadoop):
[~arpitagarwal] , sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath edited comment on HDFS-11182 at 12/3/16 11:01 AM:
[~arpitagarwal] , sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
{code}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:- LOG.info("Scheduled health check for
volume {}", reference.getVolume()); this is there only in checkAllVolumes where
as LOG.info("Checking {} volumes", references.size()); is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
was (Author: manju_hadoop):
[~Arpit Agarwal], sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath edited comment on HDFS-11182 at 12/3/16 11:00 AM:
[~Arpit Agarwal], sorry for the late code review as this comment holds good
more for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
{code}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:- LOG.info("Scheduled health check for
volume {}", reference.getVolume()); this is there only in checkAllVolumes where
as LOG.info("Checking {} volumes", references.size()); is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
was (Author: manju_hadoop):
Arpit Agarwal, sorry for the late code review as this comment holds good more
for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
[jira] [Comment Edited] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath edited comment on HDFS-11182 at 12/3/16 10:58 AM:
Arpit Agarwal, sorry for the late code review as this comment holds good more
for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
{code}
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
{code}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:- LOG.info("Scheduled health check for
volume {}", reference.getVolume()); this is there only in checkAllVolumes where
as LOG.info("Checking {} volumes", references.size()); is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
was (Author: manju_hadoop):
Arpit Agarwal, sorry for the late code review as this comment holds good more
for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
[jira] [Commented] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
[
https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717913#comment-15717913
]
Manjunath commented on HDFS-11182:
--
Arpit Agarwal, sorry for the late code review as this comment holds good more
for the JIRA HDFS-11149. Thanks for the effort for this change. In
DatasetVolumeChecker the checkAllVolumes and checkAllVolumesAsync are supposed
to do same tasks except for few additional things in checkAllVolumes. The code
can be modified such that checkAllVolumes inturn calls the checkAllVolumesAsync
and save lines of code and also make sure that both does the same intended work
without code duplication in future as well.
public ResultHandler checkAllVolumesAsync(
final FsDatasetSpi dataset,
Callback callback) {
if (timer.monotonicNow() - lastAllVolumesCheck < minDiskCheckGapMs) {
numSkippedChecks.incrementAndGet();
return null;
}
lastAllVolumesCheck = timer.monotonicNow();
final Set healthyVolumes = new HashSet<>();
final Set failedVolumes = new HashSet<>();
final Set allVolumes = new HashSet<>();
final FsDatasetSpi.FsVolumeReferences references =
dataset.getFsVolumeReferences();
final CountDownLatch latch = new CountDownLatch(references.size());
LOG.info("Checking {} volumes", references.size());
for (int i = 0; i < references.size(); ++i) {
final FsVolumeReference reference = references.getReference(i);
allVolumes.add(reference.getVolume().getStorageLocation());
// The context parameter is currently ignored.
ListenableFuture future =
delegateChecker.schedule(reference.getVolume(),
IGNORED_CONTEXT);
LOG.info("Scheduled health check for volume {}",
reference.getVolume());
Futures.addCallback(future, new ResultHandler(
reference, healthyVolumes, failedVolumes, latch, callback));
}
numAsyncDatasetChecks.incrementAndGet();
return new ResultHandler(
null, healthyVolumes, failedVolumes, latch, callback);
}
public Set checkAllVolumes(
final FsDatasetSpi dataset)
throws InterruptedException {
ResultHandler resultHandler = checkAllVolumesAsync(dataset, null);
if (resultHandler == null) {
return Collections.emptySet();
}
// Wait until our timeout elapses, after which we give up on
// the remaining volumes.
if (!resultHandler.getResultsLatch().await(maxAllowedTimeForCheckMs,
TimeUnit.MILLISECONDS)) {
LOG.warn("checkAllVolumes timed out after {} ms" +
maxAllowedTimeForCheckMs);
}
numSyncDatasetChecks.incrementAndGet();
synchronized (this) {
// All volumes that have not been detected as healthy should be
// considered failed. This is a superset of 'failedVolumes'.
//
// Make a copy under the mutex as Sets.difference() returns a view
// of a potentially changing set.
return new HashSet<>(Sets.difference(resultHandler.getAllVolumes(),
resultHandler.getHealthyVolumes()));
}
}
Although this is not a great thing but it will save duplicate code and
differences of core behaviour between the 2 methods which is currently present
as well in terms of some loggers eg:- LOG.info("Scheduled health check for
volume {}", reference.getVolume()); this is there only in checkAllVolumes where
as LOG.info("Checking {} volumes", references.size()); is present only in
checkAllVolumesAsync. Please let me know your thoughts on this.
> Update DataNode to use DatasetVolumeChecker
> ---
>
> Key: HDFS-11182
> URL: https://issues.apache.org/jira/browse/HDFS-11182
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in
> HDFS-11149 to parallelize disk checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
