[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224778 ] ASF GitHub Bot logged work on HDDS-1393: Author: ASF GitHub Bot Created on: 09/Apr/19 05:50 Start Date: 09/Apr/19 05:50 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #704: HDDS-1393. Convert all OM Bucket related operations to HA model. URL: https://github.com/apache/hadoop/pull/704#issuecomment-48738 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 28 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | +1 | mvninstall | 1116 | trunk passed | | +1 | compile | 112 | trunk passed | | +1 | checkstyle | 45 | trunk passed | | +1 | mvnsite | 93 | trunk passed | | +1 | shadedclient | 798 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 105 | trunk passed | | +1 | javadoc | 71 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 10 | Maven dependency ordering for patch | | +1 | mvninstall | 102 | the patch passed | | +1 | compile | 116 | the patch passed | | +1 | cc | 116 | the patch passed | | +1 | javac | 116 | the patch passed | | +1 | checkstyle | 24 | the patch passed | | +1 | mvnsite | 88 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 717 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 111 | the patch passed | | +1 | javadoc | 72 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 34 | common in the patch passed. | | +1 | unit | 41 | ozone-manager in the patch passed. | | -1 | unit | 1387 | integration-test in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 5125 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.om.TestScmChillMode | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/704 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 0152e5d203ff 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 69e3745 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/testReport/ | | Max. process+thread count | 5329 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/3/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224778) Time Spent: 1.5h (was: 1h 20m) > Convert all OM Bucket related operations to HA model > > >
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224777 ] ASF GitHub Bot logged work on HDDS-1393: Author: ASF GitHub Bot Created on: 09/Apr/19 05:49 Start Date: 09/Apr/19 05:49 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #704: HDDS-1393. Convert all OM Bucket related operations to HA model. URL: https://github.com/apache/hadoop/pull/704#issuecomment-48515 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 28 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 42 | Maven dependency ordering for branch | | +1 | mvninstall | 1047 | trunk passed | | +1 | compile | 108 | trunk passed | | +1 | checkstyle | 47 | trunk passed | | +1 | mvnsite | 100 | trunk passed | | +1 | shadedclient | 774 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 97 | trunk passed | | +1 | javadoc | 70 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 11 | Maven dependency ordering for patch | | +1 | mvninstall | 97 | the patch passed | | +1 | compile | 100 | the patch passed | | +1 | cc | 100 | the patch passed | | +1 | javac | 100 | the patch passed | | +1 | checkstyle | 22 | the patch passed | | +1 | mvnsite | 80 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 751 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 112 | the patch passed | | +1 | javadoc | 64 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 35 | common in the patch passed. | | +1 | unit | 37 | ozone-manager in the patch passed. | | -1 | unit | 1486 | integration-test in the patch failed. | | +1 | asflicense | 28 | The patch does not generate ASF License warnings. | | | | 5122 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.om.TestOzoneManagerHA | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/704 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 653162015ddc 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 69e3745 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/testReport/ | | Max. process+thread count | 4159 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-704/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224777) Time Spent: 1h 20m (was: 1h 10m) > Convert all OM Bucket related operations to HA model > > > Key: HDDS-1393 > URL: https://issues.apache.org/jira/browse/HDDS-1393 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813017#comment-16813017
]
Hadoop QA commented on HDFS-14403:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 8m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 5s{color} | {color:orange} root: The patch generated 4 new + 972 unchanged
- 1 fixed = 976 total (was 973) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
1s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965254/HDFS-14403.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux f3a762f2868e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Updated] (HDDS-1404) Fix typo in HDDS
[ https://issues.apache.org/jira/browse/HDDS-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bianqi updated HDDS-1404: - Attachment: HDDS-1404.1.patch > Fix typo in HDDS > > > Key: HDDS-1404 > URL: https://issues.apache.org/jira/browse/HDDS-1404 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: documentation >Affects Versions: 0.3.0 >Reporter: bianqi >Priority: Trivial > Labels: newbie > Attachments: HDDS-1404.1.patch > > > [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto#L465] > [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/hdds.proto#L172] > [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto#L37] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1404) Fix typo in HDDS
bianqi created HDDS-1404: Summary: Fix typo in HDDS Key: HDDS-1404 URL: https://issues.apache.org/jira/browse/HDDS-1404 Project: Hadoop Distributed Data Store Issue Type: Bug Components: documentation Affects Versions: 0.3.0 Reporter: bianqi [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto#L465] [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/hdds.proto#L172] [https://github.com/apache/hadoop/blob/trunk/hadoop-hdds/common/src/main/proto/StorageContainerLocationProtocol.proto#L37] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-372) There are three buffer copies in BlockOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-372: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) Thanks [~szetszwo] for the review. I have committed this change to trunk. > There are three buffer copies in BlockOutputStream > -- > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-372.001.patch, HDDS-372.002.patch, > HDDS-372.003.patch, HDDS-372.004.patch, HDDS-372.005.patch, > HDDS-372.20180829.patch > > > Currently, there are three buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > # from ByteString to ByteBuffer for checskum computation > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1008) Invalidate closed container replicas on a failed volume
[ https://issues.apache.org/jira/browse/HDDS-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-1008: --- Assignee: Supratim Deka (was: Arpit Agarwal) > Invalidate closed container replicas on a failed volume > --- > > Key: HDDS-1008 > URL: https://issues.apache.org/jira/browse/HDDS-1008 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Arpit Agarwal >Assignee: Supratim Deka >Priority: Major > > When a volume is detected as failed, all closed containers on the volume > should be marked as invalid. > Open containers will be handled separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14414) Clean up findbugs warning in branch-2
[
https://issues.apache.org/jira/browse/HDFS-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812980#comment-16812980
]
Dinesh Chitlangia commented on HDFS-14414:
--
[~jojochuang] Thanks for review and commit!
> Clean up findbugs warning in branch-2
> -
>
> Key: HDFS-14414
> URL: https://issues.apache.org/jira/browse/HDFS-14414
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Wei-Chiu Chuang
>Assignee: Dinesh Chitlangia
>Priority: Major
> Fix For: 2.10.0, 2.8.6, 2.9.3
>
> Attachments: HDFS-14414-branch-2.00.patch
>
>
> There's an unused local variable and findbugs doesn't like it.
> {noformat}
> Bug type DLS_DEAD_LOCAL_STORE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver
> In method
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream,
> INodeSymlink)
> Local variable named state
> At FSImageFormatPBINode.java:[line 623]{noformat}
> File this jira to clean it up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812953#comment-16812953
]
Akira Ajisaka commented on HDFS-14369:
--
Thank you [~ayushtkn], [~elgoiri], and [~crh]!
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch,
> HDFS-14369-HDFS-13891.005.patch, HDFS-14369-HDFS-13891.006.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812938#comment-16812938 ] qiang Liu commented on HDFS-14303: -- regernerate patch target on branch-2 > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.005.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14303: - Attachment: HDFS-14303-branch-2.005.patch > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.005.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[
https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812918#comment-16812918
]
qiang Liu edited comment on HDFS-14303 at 4/9/19 1:37 AM:
--
regernerate patch use
{code:java}
git format-patch branch-2.7
{code}
, checked line end is LF
was (Author: iamgd67):
regernerate patch use git format-patch branch-2.7, checked line end is LF
> chek block directory logic not correct when there is only meta file, print no
> meaning warn log
> --
>
> Key: HDFS-14303
> URL: https://issues.apache.org/jira/browse/HDFS-14303
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs
>Affects Versions: 2.7.3
> Environment: env free
>Reporter: qiang Liu
>Priority: Minor
> Labels: easy-fix
> Attachments: HDFS-14303-branch-2.7.001.patch,
> HDFS-14303-branch-2.7.004.patch
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> chek block directory logic not correct when there is only meta file,print no
> meaning warn log, eg:
> WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block
> ID-based layout. Actual block file path:
> /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68,
> expected block file path:
> /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812918#comment-16812918 ] qiang Liu commented on HDFS-14303: -- regernerate patch use git format-patch branch-2.7, checked line end is LF > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14303: - Attachment: (was: HDFS-14303-branch-2.7.001.patch) > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14303: - Attachment: (was: HDFS-14303-branch-2.7.3.002.patch) > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiang Liu updated HDFS-14303: - Attachment: HDFS-14303-branch-2.7.004.patch > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.004.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1383) [Ozone Upgrade] Create the project skeleton with CLI interface
[ https://issues.apache.org/jira/browse/HDDS-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812911#comment-16812911 ] Anu Engineer commented on HDDS-1383: +1, thanks for the skeleton. > [Ozone Upgrade] Create the project skeleton with CLI interface > -- > > Key: HDDS-1383 > URL: https://issues.apache.org/jira/browse/HDDS-1383 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: upgrade >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Ozone In-Place upgrade tool is a tool to upgrade hdfs data to ozone data > without data movement. > In this jira I will create a skeleton project with the cli interface without > any business logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224729=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224729
]
ASF GitHub Bot logged work on HDDS-1393:
Author: ASF GitHub Bot
Created on: 09/Apr/19 00:52
Start Date: 09/Apr/19 00:52
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #704:
HDDS-1393. Convert all OM Bucket related operations to HA model.
URL: https://github.com/apache/hadoop/pull/704#discussion_r273290389
##
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##
@@ -479,6 +479,7 @@ message InfoBucketResponse {
message SetBucketPropertyRequest {
required BucketArgs bucketArgs = 1;
+optional BucketInfo bucketInfo = 2;
Review comment:
Addressed for now as making bucketArgs as optional, so that we will not set
this during creating OMRequest in startTransaction, and we don't write
duplicate data into raft log.
Added a TODO for the suggested, as discussed offline.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 224729)
Time Spent: 1h 10m (was: 1h)
> Convert all OM Bucket related operations to HA model
>
>
> Key: HDDS-1393
> URL: https://issues.apache.org/jira/browse/HDDS-1393
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM Bucket related operations to OM HA
> model, which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like bucket not found or some
> conditions which i have not satisfied like when deleting bucket should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224724 ] ASF GitHub Bot logged work on HDDS-1393: Author: ASF GitHub Bot Created on: 09/Apr/19 00:50 Start Date: 09/Apr/19 00:50 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #704: HDDS-1393. Convert all OM Bucket related operations to HA model. URL: https://github.com/apache/hadoop/pull/704#issuecomment-481059723 Thank You @hanishakoneru for the review. I have addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224724) Time Spent: 1h (was: 50m) > Convert all OM Bucket related operations to HA model > > > Key: HDDS-1393 > URL: https://issues.apache.org/jira/browse/HDDS-1393 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In this jira, we shall convert all OM Bucket related operations to OM HA > model, which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like bucket not found or some > conditions which i have not satisfied like when deleting bucket should be > empty, these all will be executed during startTransaction, and if it fails > these requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1266) [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone
[ https://issues.apache.org/jira/browse/HDDS-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812893#comment-16812893 ] Anu Engineer commented on HDDS-1266: [~shv] I have attached the design doc for the in place upgrade. You are the first person who asked for this feature, would appreciate any feedback you might have on the design, especially if you see any issues from HDFS point of view that is not addressed. [~clayb] This is the upgrade design I had taked about in the community meetings, Please do let me know you thoughts, questions or comments. [~daryn] If you have time, I would appreciate any comments you have on the HDFS-to-Ozone in-place upgrades. Thanks Anu > [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone > > > Key: HDDS-1266 > URL: https://issues.apache.org/jira/browse/HDDS-1266 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: InPlaceUpgradesForOzone.pdf > > > This is the master JIRA to support upgrading existing HDFS clusters to have > Ozone running concurrently. One of the requirements is that we support > upgrading from HDFS to Ozone, without a full data copy. This requirement is > called "In Place upgrade", the end result of such an upgrade would be to have > the HDFS data appear in Ozone as if Ozone has taken a snap-shot of the HDFS > data. Once upgrade is complete, Ozone and HDFS will act as independent > systems. I will post a design document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1266) [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone
[ https://issues.apache.org/jira/browse/HDDS-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1266: --- Attachment: InPlaceUpgradesForOzone.pdf > [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone > > > Key: HDDS-1266 > URL: https://issues.apache.org/jira/browse/HDDS-1266 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Major > Attachments: InPlaceUpgradesForOzone.pdf > > > This is the master JIRA to support upgrading existing HDFS clusters to have > Ozone running concurrently. One of the requirements is that we support > upgrading from HDFS to Ozone, without a full data copy. This requirement is > called "In Place upgrade", the end result of such an upgrade would be to have > the HDFS data appear in Ozone as if Ozone has taken a snap-shot of the HDFS > data. Once upgrade is complete, Ozone and HDFS will act as independent > systems. I will post a design document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224679
]
ASF GitHub Bot logged work on HDDS-1393:
Author: ASF GitHub Bot
Created on: 08/Apr/19 23:38
Start Date: 08/Apr/19 23:38
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #704:
HDDS-1393. Convert all OM Bucket related operations to HA model.
URL: https://github.com/apache/hadoop/pull/704#discussion_r273278199
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3BucketManagerImpl.java
##
@@ -141,7 +141,14 @@ public void deleteS3Bucket(String bucketName) throws
IOException {
throw new OMException("No such S3 bucket. " + bucketName,
OMException.ResultCodes.S3_BUCKET_NOT_FOUND);
}
- bucketManager.deleteBucket(getOzoneVolumeName(bucketName), bucketName);
+
+ if (isRatisEnabled) {
Review comment:
Here it should be if Ratis is Enabled, we should call applyTransaction to
persist this info to DB.
As we have not separated this into 2 phase we should do this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 224679)
Time Spent: 50m (was: 40m)
> Convert all OM Bucket related operations to HA model
>
>
> Key: HDDS-1393
> URL: https://issues.apache.org/jira/browse/HDDS-1393
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM Bucket related operations to OM HA
> model, which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like bucket not found or some
> conditions which i have not satisfied like when deleting bucket should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10585) Incorrect offset/length calculation in pipeline recovery causes block corruption
[ https://issues.apache.org/jira/browse/HDFS-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-10585. Resolution: Duplicate > Incorrect offset/length calculation in pipeline recovery causes block > corruption > > > Key: HDFS-10585 > URL: https://issues.apache.org/jira/browse/HDFS-10585 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > We found incorrect offset and length calculation in pipeline recovery may > cause block corruption and results in missing blocks under a very unfortunate > scenario. > (1) A client established pipeline and started writing data to the pipeline. > (2) One of the data node in the pipeline restarted, closing the socket, and > some written data were unacknowledged. > (3) Client replaced the failed data node with a new one, initiating block > transfer to copy existing data in the block to the new datanode. > (4) The block is transferred to the new node. Crucially, the entire block, > including the unacknowledged data, was transferred. > (5) The last chunk (512 bytes) was not a full chunk, but the destination > still reserved the whole chunk in its buffer, and wrote the entire buffer to > disk, therefore some written data is garbage. > (6) When the transfer was done, the destination data node converted the > replica from temporary to rbw, which made its visible length as the length of > bytes on disk. That is to say, it thought whatever was transferred was > acknowledged. However, the visible length of the replica is different (round > up to the next multiple of 512) than the source of transfer. > (7) Client then truncated the block in the attempt to remove unacknowledged > data. However, because the visible length is equivalent of the bytes on disk, > it did not truncate unacknowledged data. > (8) When new data was appended to the destination, it skipped the bytes > already on disk. Therefore, whatever was written as garbage was not replaced. > (9) the volume scanner detected corrupt replica, but due to HDFS-10512, it > wouldn't tell NameNode to mark the replica as corrupt, so the client > continued to form a pipeline using the corrupt replica. > (10) Finally the DN that had the only healthy replica was restarted. NameNode > then update the pipeline to only contain the corrupt replica. > (11) Client continue to write to the corrupt replica, because neither client > nor the data node itself knows the replica is corrupt. When the restarted > datanodes comes back, their replica are stale, despite they are not corrupt. > Therefore, none of the replica is good and up to date. > The sequence of events was reconstructed based on DataNode/NameNode log and > my understanding of code. > Incidentally, we have observed the same sequence of events on two independent > clusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224666
]
ASF GitHub Bot logged work on HDDS-1393:
Author: ASF GitHub Bot
Created on: 08/Apr/19 23:20
Start Date: 08/Apr/19 23:20
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #704:
HDDS-1393. Convert all OM Bucket related operations to HA model.
URL: https://github.com/apache/hadoop/pull/704#discussion_r273269739
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3BucketManagerImpl.java
##
@@ -141,7 +141,14 @@ public void deleteS3Bucket(String bucketName) throws
IOException {
throw new OMException("No such S3 bucket. " + bucketName,
OMException.ResultCodes.S3_BUCKET_NOT_FOUND);
}
- bucketManager.deleteBucket(getOzoneVolumeName(bucketName), bucketName);
+
+ if (isRatisEnabled) {
Review comment:
Here it should be !isRatisEnabled
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 224666)
Time Spent: 0.5h (was: 20m)
> Convert all OM Bucket related operations to HA model
>
>
> Key: HDDS-1393
> URL: https://issues.apache.org/jira/browse/HDDS-1393
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM Bucket related operations to OM HA
> model, which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like bucket not found or some
> conditions which i have not satisfied like when deleting bucket should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1393) Convert all OM Bucket related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1393?focusedWorklogId=224667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224667
]
ASF GitHub Bot logged work on HDDS-1393:
Author: ASF GitHub Bot
Created on: 08/Apr/19 23:20
Start Date: 08/Apr/19 23:20
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #704:
HDDS-1393. Convert all OM Bucket related operations to HA model.
URL: https://github.com/apache/hadoop/pull/704#discussion_r273271723
##
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##
@@ -479,6 +479,7 @@ message InfoBucketResponse {
message SetBucketPropertyRequest {
required BucketArgs bucketArgs = 1;
+optional BucketInfo bucketInfo = 2;
Review comment:
BucketArgs and BucketInfo have a lot of common fields. Can we combine them
into one proto?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 224667)
Time Spent: 40m (was: 0.5h)
> Convert all OM Bucket related operations to HA model
>
>
> Key: HDDS-1393
> URL: https://issues.apache.org/jira/browse/HDDS-1393
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM Bucket related operations to OM HA
> model, which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like bucket not found or some
> conditions which i have not satisfied like when deleting bucket should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1388) Add a shell script to run MiniOzoneChaosCluster using mvn exec
[ https://issues.apache.org/jira/browse/HDDS-1388?focusedWorklogId=224648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224648 ] ASF GitHub Bot logged work on HDDS-1388: Author: ASF GitHub Bot Created on: 08/Apr/19 22:41 Start Date: 08/Apr/19 22:41 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #709: HDDS-1388. Add a shell script to run MiniOzoneChaosCluster using mvn exec. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/709#issuecomment-48103 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 31 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | +1 | mvninstall | 1018 | trunk passed | | +1 | compile | 1005 | trunk passed | | +1 | checkstyle | 193 | trunk passed | | +1 | mvnsite | 113 | trunk passed | | +1 | shadedclient | 695 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 50 | trunk passed | | +1 | javadoc | 59 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 66 | the patch passed | | +1 | compile | 1017 | the patch passed | | +1 | javac | 1017 | the patch passed | | +1 | checkstyle | 200 | the patch passed | | +1 | mvnsite | 82 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | shelldocs | 29 | There were no new shelldocs issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 647 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 57 | the patch passed | | +1 | javadoc | 59 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 108 | server-scm in the patch passed. | | -1 | unit | 2441 | integration-test in the patch failed. | | +1 | asflicense | 64 | The patch does not generate ASF License warnings. | | | | 8270 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-709/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/709 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle shellcheck shelldocs | | uname | Linux 54a32adbbf6a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 69e3745 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | shellcheck | v0.4.6 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-709/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-709/1/testReport/ | | Max. process+thread count | 4174 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-709/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224648) Time Spent: 20m (was: 10m) > Add a shell script to run MiniOzoneChaosCluster using mvn exec > -- > > Key: HDDS-1388 > URL:
[jira] [Updated] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Gregorian updated HDFS-14403: - Attachment: HDFS-14403.002.patch > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > Attachments: CostBasedFairCallQueueDesign_v0.pdf, > HDFS-14403.001.patch, HDFS-14403.002.patch > > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812852#comment-16812852
]
Christopher Gregorian commented on HDFS-14403:
--
Thanks for the comments and suggestions! Patch 002 should fix up those tests
and the checkstyle warnings, as well as address [~xkrogen]'s suggestions: I
built upon those by switching to using a {{LongAdder}} in the internal
{{UserLockHoldTime}} class instead of an {{AtomicLong}}, and chose to use 5 as
the default write time multiplier.
I approached [~linyiqun]'s second suggestion by changing the
{{DecayRpcProvider}} to update from the {{CostProvider}} on each call only when
there isn't a value in the cache, and keeping the update on decay sweep: I
believe this is a good compromise in minimizing contention on the
{{FSNamesystemLock}}'s {{userLockTimes}} while still keeping the totals up to
date for new users. Let me know if there's a reason I'm not thinking of where
updating every call is preferable :)
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf, HDFS-14403.001.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14384) When lastLocatedBlock token expire, it will take 1~3s second to refetch it.
[
https://issues.apache.org/jira/browse/HDFS-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812842#comment-16812842
]
Daryn Sharp commented on HDFS-14384:
The {{LocatedBlocks}} class appears fundamentally broken – a mutable located
block list but an immutable last block, file size, etc. Ignoring that, I'm
initially confused by:
{code}
locatedBlocks.insertRange(targetBlockIdx,
newBlocks.getLocatedBlocks());
+
+// Update the LastLocatedBlock, if offset is for last block.
+if (offset >= locatedBlocks.getFileLength()) {
+ locatedBlocks = new LocatedBlocks(newBlocks.getFileLength(), .
{code}
# Why go through the work of inserting the new blocks in the old block list
when it's going to be thrown away?
# Why recreate locatedBlocks from newBlocks instead of just assigning newBlocks
to locatedBlocks?
> When lastLocatedBlock token expire, it will take 1~3s second to refetch it.
> ---
>
> Key: HDFS-14384
> URL: https://issues.apache.org/jira/browse/HDFS-14384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14384.001.patch
>
>
> Scenario :
> 1. Write file with one block which is in-progress.
> 2. Open input stream and close the output stream.
> 3. Wait for block token expiration and read the data.
> 4. Last block read take 1~3 sec to read it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812813#comment-16812813 ] Daryn Sharp commented on HDFS-14418: I agree the super user check should be done once and should be done w/o the lock. I worry about lots of things but I'm not concerned about the ultra rare case of a call that lost superuser privs during the lock wait. Either a rpc to refresh the groups was sent during lock wait, which shouldn't have barged in front of the pending call anyway, or the group cache just so happened to expire during the lock wait. Agree that we better ensure there's adequate test coverage for permissions. > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14418-01.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14384) When lastLocatedBlock token expire, it will take 1~3s second to refetch it.
[
https://issues.apache.org/jira/browse/HDFS-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812763#comment-16812763
]
Hadoop QA commented on HDFS-14384:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
55 unchanged - 0 fixed = 57 total (was 55) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14384 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965131/HDFS-14384.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ba9e8f8d3bd3 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk
[jira] [Work logged] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?focusedWorklogId=224574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224574 ] ASF GitHub Bot logged work on HDDS-1401: Author: ASF GitHub Bot Created on: 08/Apr/19 20:15 Start Date: 08/Apr/19 20:15 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #708: HDDS-1401. Static ContainerCache in Datanodes can result in overwrite of container db. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/708#issuecomment-480989978 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 27 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for branch | | +1 | mvninstall | 1024 | trunk passed | | +1 | compile | 968 | trunk passed | | +1 | checkstyle | 180 | trunk passed | | +1 | mvnsite | 121 | trunk passed | | +1 | shadedclient | 991 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 95 | trunk passed | | +1 | javadoc | 83 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 22 | Maven dependency ordering for patch | | +1 | mvninstall | 94 | the patch passed | | +1 | compile | 906 | the patch passed | | +1 | javac | 906 | the patch passed | | +1 | checkstyle | 181 | the patch passed | | +1 | mvnsite | 103 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 615 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | -1 | findbugs | 59 | hadoop-hdds/container-service generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | javadoc | 88 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 73 | container-service in the patch failed. | | +1 | unit | 109 | server-scm in the patch passed. | | -1 | unit | 1382 | integration-test in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 7190 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdds/container-service | | | org.apache.hadoop.ozone.container.common.utils.ContainerCache.removeLRU(AbstractLinkedMap$LinkEntry) does not release lock on all exception paths At ContainerCache.java:on all exception paths At ContainerCache.java:[line 110] | | Failed junit tests | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/708 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6cfaff448837 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 69e3745 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/new-findbugs-hadoop-hdds_container-service.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/patch-unit-hadoop-hdds_container-service.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/testReport/ | | Max. process+thread count | 4009 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
[jira] [Commented] (HDDS-372) There are three buffer copies in BlockOutputStream
[ https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812738#comment-16812738 ] Shashikant Banerjee commented on HDDS-372: -- The test failures reported are not related to the patch. The client test failures reported should be addressed with HDDS-1373. > There are three buffer copies in BlockOutputStream > -- > > Key: HDDS-372 > URL: https://issues.apache.org/jira/browse/HDDS-372 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Tsz Wo Nicholas Sze >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-372.001.patch, HDDS-372.002.patch, > HDDS-372.003.patch, HDDS-372.004.patch, HDDS-372.005.patch, > HDDS-372.20180829.patch > > > Currently, there are three buffer copies in ChunkOutputStream > # from byte[] to ByteBuffer, and > # from ByteBuffer to ByteString. > # from ByteString to ByteBuffer for checskum computation > We should eliminate the ByteBuffer in the middle. > For zero copy io, we should support WritableByteChannel instead of > OutputStream. It won't be done in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1395) Key write fails with "BlockOutputStream has been closed"
[
https://issues.apache.org/jira/browse/HDDS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-1395:
--
Attachment: HDDS-1395.000.patch
> Key write fails with "BlockOutputStream has been closed"
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster
> Attachments: HDDS-1395.000.patch
>
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1395) Key write fails with "BlockOutputStream has been closed"
[
https://issues.apache.org/jira/browse/HDDS-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDDS-1395:
-
Assignee: Shashikant Banerjee
> Key write fails with "BlockOutputStream has been closed"
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1388) Add a shell script to run MiniOzoneChaosCluster using mvn exec
[ https://issues.apache.org/jira/browse/HDDS-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1388: Status: Patch Available (was: Open) > Add a shell script to run MiniOzoneChaosCluster using mvn exec > -- > > Key: HDDS-1388 > URL: https://issues.apache.org/jira/browse/HDDS-1388 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This jira adds a shell script to run MiniOzoneChaosCluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-1376:
Labels: MiniOzoneChaosCluster (was: )
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Priority: Major
> Labels: MiniOzoneChaosCluster
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[
https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812698#comment-16812698
]
Íñigo Goiri commented on HDFS-14418:
My first goal is to make sure that we have coverage to make sure we are
checking for the permissions.
Can you also give a path for each method explaining where the permissions are
checked?
Regarding the one that checks first outside of the lock and then again, we
could have a case where this breaks.
For example, we start {{createEncryptionZone()}}, check the permissions and
then wait for the lock.
At this time the permissions can change and then when we go into the lock, we
have the wrong ones.
I agree that is not that key but we should get feedback from the people to
wrote this to see the implications.
> Remove redundant super user priveledge checks from namenode.
>
>
> Key: HDFS-14418
> URL: https://issues.apache.org/jira/browse/HDFS-14418
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14418-01.patch
>
>
> There are couple of methods that unnecessarily double checks super user
> privileged at namenode, which can reduced to single.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1388) Add a shell script to run MiniOzoneChaosCluster using mvn exec
[ https://issues.apache.org/jira/browse/HDDS-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1388: - Labels: pull-request-available (was: ) > Add a shell script to run MiniOzoneChaosCluster using mvn exec > -- > > Key: HDDS-1388 > URL: https://issues.apache.org/jira/browse/HDDS-1388 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > > This jira adds a shell script to run MiniOzoneChaosCluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1388) Add a shell script to run MiniOzoneChaosCluster using mvn exec
[ https://issues.apache.org/jira/browse/HDDS-1388?focusedWorklogId=224536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224536 ] ASF GitHub Bot logged work on HDDS-1388: Author: ASF GitHub Bot Created on: 08/Apr/19 18:41 Start Date: 08/Apr/19 18:41 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #709: HDDS-1388. Add a shell script to run MiniOzoneChaosCluster using mvn exec. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/709 This jira adds a shell script to run MiniOzoneChaosCluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224536) Time Spent: 10m Remaining Estimate: 0h > Add a shell script to run MiniOzoneChaosCluster using mvn exec > -- > > Key: HDDS-1388 > URL: https://issues.apache.org/jira/browse/HDDS-1388 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.5.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This jira adds a shell script to run MiniOzoneChaosCluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1403) KeyOutputStream writes fails after max retries while writing to a closed container
Mukul Kumar Singh created HDDS-1403: --- Summary: KeyOutputStream writes fails after max retries while writing to a closed container Key: HDDS-1403 URL: https://issues.apache.org/jira/browse/HDDS-1403 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.4.0 Reporter: Mukul Kumar Singh Currently a Ozone Client retries a write operation 5 times. It is possible that the container being written to is already closed by the time it is written to. The key write will fail after retrying multiple times with this error. This needs to be fixed as this is an internal error. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812684#comment-16812684
]
Hadoop QA commented on HDDS-1301:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
34s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdds: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-ozone: The patch generated 24 new + 0
unchanged - 0 fixed = 24 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 24s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.om.exceptions.TestResultCodes |
\\
\\
|| Subsystem ||
[jira] [Work logged] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?focusedWorklogId=224515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224515 ] ASF GitHub Bot logged work on HDDS-1402: Author: ASF GitHub Bot Created on: 08/Apr/19 18:14 Start Date: 08/Apr/19 18:14 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #707: HDDS-1402. Remove unused ScmBlockLocationProtocol from ObjectStoreHandler URL: https://github.com/apache/hadoop/pull/707#issuecomment-480944963 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 26 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1005 | trunk passed | | +1 | compile | 48 | trunk passed | | +1 | checkstyle | 18 | trunk passed | | +1 | mvnsite | 26 | trunk passed | | +1 | shadedclient | 667 | branch has no errors when building and testing our client artifacts. | | +1 | findbugs | 38 | trunk passed | | +1 | javadoc | 26 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 41 | the patch passed | | +1 | compile | 18 | the patch passed | | +1 | javac | 18 | the patch passed | | +1 | checkstyle | 11 | the patch passed | | +1 | mvnsite | 22 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 710 | patch has no errors when building and testing our client artifacts. | | -1 | findbugs | 44 | hadoop-ozone/objectstore-service generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | javadoc | 19 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 23 | objectstore-service in the patch passed. | | +1 | asflicense | 30 | The patch does not generate ASF License warnings. | | | | 2876 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-ozone/objectstore-service | | | Dead store to scmBlockAddress in new org.apache.hadoop.hdfs.server.datanode.ObjectStoreHandler(Configuration) At ObjectStoreHandler.java:new org.apache.hadoop.hdfs.server.datanode.ObjectStoreHandler(Configuration) At ObjectStoreHandler.java:[line 106] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-707/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/707 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 0d186edb2f23 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 69e3745 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-707/1/artifact/out/new-findbugs-hadoop-ozone_objectstore-service.html | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-707/1/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/objectstore-service U: hadoop-ozone/objectstore-service | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-707/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224515) Time Spent: 20m (was: 10m) > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When I
[jira] [Commented] (HDDS-1340) Add List Containers API for Recon
[ https://issues.apache.org/jira/browse/HDDS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812669#comment-16812669 ] Hudson commented on HDDS-1340: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16363 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16363/]) HDDS-1340. Add List Containers API for Recon (#648) (bharat: rev 69e3745b86aaa85e7d34dc41b5bbf2561c226f7a) * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerKeyService.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/ContainerDBServiceProvider.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ContainerDBServiceProviderImpl.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerKeyService.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ContainerMetadata.java * (delete) hadoop-ozone/ozone-recon/src/main/resources/webapps.recon.WEB-INF/web.xml * (add) hadoop-ozone/ozone-recon/src/main/resources/webapps/recon/WEB-INF/web.xml > Add List Containers API for Recon > - > > Key: HDDS-1340 > URL: https://issues.apache.org/jira/browse/HDDS-1340 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Recon server should support "/containers" API that lists all the containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812663#comment-16812663 ] Ayush Saxena commented on HDFS-14418: - Thanx [~elgoiri] for the comment. I think technically with removal of redundant super user privileged check, there is no functionality change, that will any how check the permission as it was earlier, In case of failure, it would fail at first only and in the correct case it was checking twice.Now that will check only once. I guess that won't have any functionality impact. I am pretty sure that was for no good. And no clue how that got it.That is just 2-3 lines below. So we aren't changing any behavior, that should be fine I guess!!! That only hampered the performance.Isn't doing any good.User can't change in 2 lines. Regarding the redundant write lock, there is checking before taking write lock and even after, and this behavior is at a lot of places (I guess all EC commands too). Might be to ensure even after taking write lock, The operation is supported. Pls Review > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14418-01.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1401: Status: Patch Available (was: Open) > Static ContainerCache in Datanodes can result in overwrite of container db > -- > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The datanodes use a static Container cache, and this container cache is key'd > using the container ID. > In a MiniOzoneCluster environment, this can lead to multiple container on > different datanodes to use the same rocksdb leading to inconsistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1401: Description: The datanodes use a static Container cache, and this container cache is key'd using the container ID. In a MiniOzoneCluster environment, this can lead to multiple container on different datanodes to use the same rocksdb leading to inconsistency. was: Key Read fails with Unable to find the block NO_SUCH_BLOCK after reducing the value of OZONE_CONTAINER_CACHE_SIZE. The reads are tried on the other datanodes but it failed on all the 3 datanodes. > Static ContainerCache in Datanodes can result in overwrite of container db > -- > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster > > The datanodes use a static Container cache, and this container cache is key'd > using the container ID. > In a MiniOzoneCluster environment, this can lead to multiple container on > different datanodes to use the same rocksdb leading to inconsistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1401: - Labels: MiniOzoneChaosCluster pull-request-available (was: MiniOzoneChaosCluster) > Static ContainerCache in Datanodes can result in overwrite of container db > -- > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster, pull-request-available > > The datanodes use a static Container cache, and this container cache is key'd > using the container ID. > In a MiniOzoneCluster environment, this can lead to multiple container on > different datanodes to use the same rocksdb leading to inconsistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?focusedWorklogId=224501=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224501 ] ASF GitHub Bot logged work on HDDS-1401: Author: ASF GitHub Bot Created on: 08/Apr/19 17:56 Start Date: 08/Apr/19 17:56 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #708: HDDS-1401. Static ContainerCache in Datanodes can result in overwrite of container db. Contributed by Mukul Kumar Singh. URL: https://github.com/apache/hadoop/pull/708 The datanodes use a static Container cache, and this container cache is key'd using the container ID. In a MiniOzoneCluster environment, this can lead to multiple container on different datanodes to use the same rocksdb leading to inconsistency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224501) Time Spent: 10m Remaining Estimate: 0h > Static ContainerCache in Datanodes can result in overwrite of container db > -- > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The datanodes use a static Container cache, and this container cache is key'd > using the container ID. > In a MiniOzoneCluster environment, this can lead to multiple container on > different datanodes to use the same rocksdb leading to inconsistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14414) Clean up findbugs warning in branch-2
[
https://issues.apache.org/jira/browse/HDFS-14414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14414:
---
Resolution: Fixed
Fix Version/s: 2.9.3
2.8.6
2.10.0
Status: Resolved (was: Patch Available)
Pushed to branch-2, branch-2.9 and branch-2.8. Thanks [~dineshchitlangia]!
> Clean up findbugs warning in branch-2
> -
>
> Key: HDFS-14414
> URL: https://issues.apache.org/jira/browse/HDFS-14414
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Wei-Chiu Chuang
>Assignee: Dinesh Chitlangia
>Priority: Major
> Fix For: 2.10.0, 2.8.6, 2.9.3
>
> Attachments: HDFS-14414-branch-2.00.patch
>
>
> There's an unused local variable and findbugs doesn't like it.
> {noformat}
> Bug type DLS_DEAD_LOCAL_STORE (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver
> In method
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(OutputStream,
> INodeSymlink)
> Local variable named state
> At FSImageFormatPBINode.java:[line 623]{noformat}
> File this jira to clean it up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1401) Static ContainerCache in Datanodes can result in overwrite of container db
[ https://issues.apache.org/jira/browse/HDDS-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1401: Summary: Static ContainerCache in Datanodes can result in overwrite of container db (was: Key Read fails with Unable to find the block, after reducing the size of container cache) > Static ContainerCache in Datanodes can result in overwrite of container db > -- > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster > > Key Read fails with Unable to find the block NO_SUCH_BLOCK after reducing the > value of OZONE_CONTAINER_CACHE_SIZE. > The reads are tried on the other datanodes but it failed on all the 3 > datanodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[
https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812653#comment-16812653
]
Hadoop QA commented on HDFS-14418:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14418 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965204/HDFS-14418-01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0214070a9676 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bb8dda2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26594/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26594/testReport/ |
| Max. process+thread count | 3923 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Updated] (HDFS-14384) When lastLocatedBlock token expire, it will take 1~3s second to refetch it.
[ https://issues.apache.org/jira/browse/HDFS-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-14384: -- Status: Patch Available (was: Open) > When lastLocatedBlock token expire, it will take 1~3s second to refetch it. > --- > > Key: HDFS-14384 > URL: https://issues.apache.org/jira/browse/HDFS-14384 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.7.2 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-14384.001.patch > > > Scenario : > 1. Write file with one block which is in-progress. > 2. Open input stream and close the output stream. > 3. Wait for block token expiration and read the data. > 4. Last block read take 1~3 sec to read it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1340) Add List Containers API for Recon
[ https://issues.apache.org/jira/browse/HDDS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812580#comment-16812580 ] Bharat Viswanadham commented on HDDS-1340: -- Thank You [~vivekratnavel] for the contribution and [~linyiqun] for the review. I have committed this to trunk. > Add List Containers API for Recon > - > > Key: HDDS-1340 > URL: https://issues.apache.org/jira/browse/HDDS-1340 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Recon server should support "/containers" API that lists all the containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1340) Add List Containers API for Recon
[ https://issues.apache.org/jira/browse/HDDS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1340: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) > Add List Containers API for Recon > - > > Key: HDDS-1340 > URL: https://issues.apache.org/jira/browse/HDDS-1340 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Recon server should support "/containers" API that lists all the containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1340) Add List Containers API for Recon
[ https://issues.apache.org/jira/browse/HDDS-1340?focusedWorklogId=224452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224452 ] ASF GitHub Bot logged work on HDDS-1340: Author: ASF GitHub Bot Created on: 08/Apr/19 16:36 Start Date: 08/Apr/19 16:36 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #648: HDDS-1340. Add List Containers API for Recon URL: https://github.com/apache/hadoop/pull/648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224452) Time Spent: 4h 40m (was: 4.5h) > Add List Containers API for Recon > - > > Key: HDDS-1340 > URL: https://issues.apache.org/jira/browse/HDDS-1340 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Recon server should support "/containers" API that lists all the containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1340) Add List Containers API for Recon
[ https://issues.apache.org/jira/browse/HDDS-1340?focusedWorklogId=224449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224449 ] ASF GitHub Bot logged work on HDDS-1340: Author: ASF GitHub Bot Created on: 08/Apr/19 16:35 Start Date: 08/Apr/19 16:35 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #648: HDDS-1340. Add List Containers API for Recon URL: https://github.com/apache/hadoop/pull/648#issuecomment-480906505 Test failures and acceptance test failures are not related to this patch. HDDS-1390 fixed S3 acceptance test failures. @vivekratnavel confirmed that this branch is not rebased with the trunk which has HDDS-1390. I will commit this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224449) Time Spent: 4.5h (was: 4h 20m) > Add List Containers API for Recon > - > > Key: HDDS-1340 > URL: https://issues.apache.org/jira/browse/HDDS-1340 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Vivek Ratnavel Subramanian >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Recon server should support "/containers" API that lists all the containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1401) Key Read fails with Unable to find the block, after reducing the size of container cache
[ https://issues.apache.org/jira/browse/HDDS-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDDS-1401: --- Assignee: Mukul Kumar Singh (was: Shashikant Banerjee) > Key Read fails with Unable to find the block, after reducing the size of > container cache > > > Key: HDDS-1401 > URL: https://issues.apache.org/jira/browse/HDDS-1401 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Blocker > Labels: MiniOzoneChaosCluster > > Key Read fails with Unable to find the block NO_SUCH_BLOCK after reducing the > value of OZONE_CONTAINER_CACHE_SIZE. > The reads are tried on the other datanodes but it failed on all the 3 > datanodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1402: - Labels: pull-request-available (was: ) > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?focusedWorklogId=224446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224446 ] ASF GitHub Bot logged work on HDDS-1402: Author: ASF GitHub Bot Created on: 08/Apr/19 16:28 Start Date: 08/Apr/19 16:28 Worklog Time Spent: 10m Work Description: elek commented on pull request #707: HDDS-1402. Remove unused ScmBlockLocationProtocol from ObjectStoreHandler URL: https://github.com/apache/hadoop/pull/707 When I analyzed the usages of the available RPC protocols in Ozone I found that the ScmBlockLocationProtocol is not used in ObjectStore at all. I would propose to remove it... See: https://issues.apache.org/jira/browse/HDDS-1402 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224446) Time Spent: 10m Remaining Estimate: 0h > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
[ https://issues.apache.org/jira/browse/HDDS-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-1402: --- Status: Patch Available (was: Open) > Remove unused ScmBlockLocationProtocol from ObjectStoreHandler > -- > > Key: HDDS-1402 > URL: https://issues.apache.org/jira/browse/HDDS-1402 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When I analyzed the usages of the available RPC protocols in Ozone I found > that the ScmBlockLocationProtocol is not used in ObjectStore at all. > I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1402) Remove unused ScmBlockLocationProtocol from ObjectStoreHandler
Elek, Marton created HDDS-1402: -- Summary: Remove unused ScmBlockLocationProtocol from ObjectStoreHandler Key: HDDS-1402 URL: https://issues.apache.org/jira/browse/HDDS-1402 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Elek, Marton Assignee: Elek, Marton When I analyzed the usages of the available RPC protocols in Ozone I found that the ScmBlockLocationProtocol is not used in ObjectStore at all. I would propose to remove it... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812567#comment-16812567
]
Erik Krogen commented on HDFS-14403:
Hey [~linyiqun], regarding:
{quote}
Can we refactor LockCostProvider into COMMON not in HDFS module? Based on the
current approach, we maintain a new mapping in FSN lock but it's actual used by
FCQ not for HDFS itself. I mean for HDFS, it can do more easier, pass the rpc
lock time into FCQ. Then FCQ implement plugin cost provider. Isn't this a
better way?
{quote}
I'm not too sure about this. Would your proposal be that {{LockCostProvider}}
lives within common, and exposes methods like {{incrementUserReadHoldTime}} and
{{incrementUserWriteHoldTime}}, which would be called by {{FSNamesystemLock}}?
The only advantage I see to this is that the logic becomes re-usable by other
modules, but the idea that the cost of an operation is equal to its lock hold
time is HDFS-specific, so it seems to me that it makes more sense to keep this
isolated within the HDFS module. Please let me know if I misunderstood you or
if you disagree.
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf, HDFS-14403.001.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14369:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-13891
Status: Resolved (was: Patch Available)
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch,
> HDFS-14369-HDFS-13891.005.patch, HDFS-14369-HDFS-13891.006.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812557#comment-16812557
]
Ayush Saxena commented on HDFS-14369:
-
Committed.
Thanks [~ajisakaa] for the Contribution, [~crh] for the report and [~elgoiri]
for the review.
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch,
> HDFS-14369-HDFS-13891.005.patch, HDFS-14369-HDFS-13891.006.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812546#comment-16812546
]
Erik Krogen commented on HDFS-14403:
[~cgregori], thanks for getting us started with the v001 patch! Besides fixing
the checkstyle and test failures (at least TestHdfsConfigFields and TestRPC
look related, you should check the others also), I have a few comments:
* {{DecayRpcScheduler#decayCurrentCounts()}} -- you changed this from
{{private}} to {{package-private}}, should it have a {{@VisibleForTesting}}
annotation?
* For {{DecayRpcScheduler#getCallCost()}}, now seems like a good time to remove
the {{throws InterruptedException}} (it doesn't) and to fix the Javadoc
formatting by adding a blank line before the {{@param}} tag.
* The Javadoc change for {{computePriorityLevel()}} is a bit confusing; it made
me think that the {{cost}} argument was the total cost across all users'
operations (i.e. {{totalDecayedCallCost}}). Can you make it more clear that it
is the total cost for a specific user?
* It doesn't seem that {{getPriorityLevelAndUpdate()}} is a useful function to
have just for testing, when a test could call the two existing methods
{{getPriorityLevel()}} and {{updateFromCostProvider()}}. Can we remove it?
* I think a default value of {{DFS_NAMENODE_LOCKCOSTPROVIDER_WRITEMULTIPLIER}}
of higher than 1 probably makes sense, maybe 5 or 10?
* The two new methods on {{FSNamesystem}} that don't have Javadoc should get
it. It would also be helpful if you could add Javadoc to the
{{FSNamesystemLock#userLockTimes}} field explaining how it works.
* I'm not sure if I like the method name {{removeUserLockTimes()}}, maybe
something like {{stopTrackingUserLockTimes()}} would be better?
* Within {{FSNamesystemLock}}, the lock hold times are stored within a
{{List}}. I know this is how it's done within
{{DecayRpcScheduler}}, but I'm not really a fan, as it's confusing upon first
glance about what it actually contains. My personal preference would be to
create a private inner class that has two {{AtomicLong}} fields, but let me
know if you disagree.
* For the {{addCallQueueConfigurationCallback}} method, since we are on Java 8,
we should take advantage of the built-in {{Consumer}} functional interface
rather than using Guava.
* Within the callback created within {{NameNodeRpcServer}}, I'm thinking we
should probably call {{namesystem.setUserLockTimeCollection(false)}} if the new
{{scheduler}} does not have a {{LockCostProvider}} so that the collection will
be disabled if the RPC scheduler is no longer using this.
* For the {{testUser(Read|Write)HoldTimes}} tests, I think the {{spy}} makes
sense to mock out multiple users. But for the other two tests, with only one
user, can we avoid the spy by just saying {{user1 =
fsLock.getCurrentUsername()}} ?
[~starphin] - thanks for taking a look! Are you requesting information on what
the total QPS for the abusive user (1000 entry listStatus) and the non-abusive
users (1 entry listStatus or mkdir) was in each of the benchmarks? I think we
should be able to supply this -- [~cgregori] we should be able to collect these
from the Dynamometer results?
{quote}
Is it possible that scheduler with LockCostProvider schedules more low cost
operations like listStatus on a directory with one subdirectories, which
results in a lower queue time.
{quote}
Yes, I agree that this is why the queue time goes down, and it is part of the
intended effect: the expensive operation is throttled, in favor of the lower
cost operations.
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf, HDFS-14403.001.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812545#comment-16812545
]
Íñigo Goiri commented on HDFS-13248:
Thanks [~hexiaoqiao] for clarifying the {{ClientProtocol}} scope.
It makes sense to just use the new one between the Router and the Namenode.
However, I'm not a fan of duplicating methods and having extra signatures for
similar things.
My vote is for modifying the IPC/RPC layer to add the extra clientHostname
field.
The auditing that [~ayushtkn] mentioned is also a plus.
The concerns regarding this approach are:
* Fake client hostname: I don't quite get this one.
* Security vulnerabilities: I think this should be fine.
* Common core IPC layer protocol changes: I think we can handle this.
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812543#comment-16812543
]
Yiqun Lin commented on HDFS-14403:
--
Thanks for working on this, [~cgregori] and [~xkrogen]. I took some time to
read the design and implementation in patch. Mainly two comments:
Can we refactor LockCostProvider into COMMON not in HDFS module? Based on the
current approach, we maintain a new mapping in FSN lock but it's actual used by
FCQ not for HDFS itself. I mean for HDFS, it can do more easier, pass the rpc
lock time into FCQ. Then FCQ implement plugin cost provider. Isn't this a
better way?
{quote}
In DecayRpcScheduler, recordOperation is called in getPriorityLevel (once
before each call) and getAndResetCost is called upon each decay sweep (every
few seconds, typically).
{quote}
We can make getAndResetCost updated more faster, don't must each decay
interval. The original way should still make sense.
The improved way:
{code}
/**
- * Update the scheduleCache to match current conditions in callCounts.
+ * Queries the CostProvider for the cumulative cost since the last call for
+ * each user that currently has a record in callCosts.
+ */
+ void updateFromCostProvider(Object identity) {
+long totalDecayedCostIncrease = 0;
+long totalRawCostIncrease = 0;
+
+ String identity = entry.getKey().toString();
+ long newCost = costProvider.getAndResetCost(identity);
+
+ AtomicLong decayedCost = entry.getValue().get(0);
+ decayedCost.getAndAdd(newCost);
+ AtomicLong rawCost = entry.getValue().get(1);
+ rawCost.getAndAdd(newCost);
+
+ totalDecayedCostIncrease += newCost;
+ totalRawCostIncrease += newCost;
+
+totalDecayedCallCost.getAndAdd(totalDecayedCostIncrease);
+totalRawCallCost.getAndAdd(totalRawCostIncrease);
+ }
{code}
Do the update operation when each request is coming. Since every time we only
updated for specified user, the cost of this should be okay.
{code}
.. private int cachedOrComputedPriorityLevel(Object identity) {
try {
updateFromCostProvider(identity);
,,,
{code}
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf, HDFS-14403.001.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1301) Optimize recursive ozone filesystem apis
[ https://issues.apache.org/jira/browse/HDDS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDDS-1301: -- Status: Patch Available (was: Open) > Optimize recursive ozone filesystem apis > > > Key: HDDS-1301 > URL: https://issues.apache.org/jira/browse/HDDS-1301 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-1301.001.patch > > > This Jira aims to optimise recursive apis in ozone file system. These are the > apis which have a recursive flag which requires an operation to be performed > on all the children of the directory. The Jira would add support for > recursive apis in Ozone manager in order to reduce the number of rpc calls to > Ozone Manager. Also currently these operations are not atomic. This Jira > would make all the operations in ozone filesystem atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1301) Optimize recursive ozone filesystem apis
[ https://issues.apache.org/jira/browse/HDDS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812538#comment-16812538 ] Lokesh Jain commented on HDDS-1301: --- Uploaded v1 patch for review. Will create pull request for the same. > Optimize recursive ozone filesystem apis > > > Key: HDDS-1301 > URL: https://issues.apache.org/jira/browse/HDDS-1301 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-1301.001.patch > > > This Jira aims to optimise recursive apis in ozone file system. These are the > apis which have a recursive flag which requires an operation to be performed > on all the children of the directory. The Jira would add support for > recursive apis in Ozone manager in order to reduce the number of rpc calls to > Ozone Manager. Also currently these operations are not atomic. This Jira > would make all the operations in ozone filesystem atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1301) Optimize recursive ozone filesystem apis
[ https://issues.apache.org/jira/browse/HDDS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDDS-1301: -- Attachment: HDDS-1301.001.patch > Optimize recursive ozone filesystem apis > > > Key: HDDS-1301 > URL: https://issues.apache.org/jira/browse/HDDS-1301 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-1301.001.patch > > > This Jira aims to optimise recursive apis in ozone file system. These are the > apis which have a recursive flag which requires an operation to be performed > on all the children of the directory. The Jira would add support for > recursive apis in Ozone manager in order to reduce the number of rpc calls to > Ozone Manager. Also currently these operations are not atomic. This Jira > would make all the operations in ozone filesystem atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[
https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812535#comment-16812535
]
Íñigo Goiri commented on HDFS-14418:
The case for {{createEncryptionZone()}} is specially weird.
Lines 7399 and 7400 do the same as 7405 and 7406 outside the lock.
Any idea what is the idea there?
BTW, is the test coverage good to verify that the permissions are checked?
> Remove redundant super user priveledge checks from namenode.
>
>
> Key: HDFS-14418
> URL: https://issues.apache.org/jira/browse/HDFS-14418
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14418-01.patch
>
>
> There are couple of methods that unnecessarily double checks super user
> privileged at namenode, which can reduced to single.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812511#comment-16812511 ] Erik Krogen commented on HDFS-14406: I'd like to echo [~daryn]'s sentiment that this probably won't provide much useful information for HDFS given the lock behavior. You can see more discussion of this in HDFS-14403, a recently updated proposal to extend the FairCallQueue to consider cost of an operation (in the case of HDFS, this is considered to be lock hold time). The design document has some extended discussion of why looking at the overall processing time is insufficient for HDFS. In that patch, we've surfaced information about the lock hold time itself up to the RPC layer; maybe it would make more sense to surface that value rather than the processing time? [~daryn], I'd love to learn more about your patch for excluding lock wait time from processing time. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812508#comment-16812508
]
Íñigo Goiri commented on HDFS-14369:
+1 on [^HDFS-14369-HDFS-13891.006.patch].
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch,
> HDFS-14369-HDFS-13891.005.patch, HDFS-14369-HDFS-13891.006.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812475#comment-16812475
]
Ayush Saxena commented on HDFS-14369:
-
Thanx [~ajisakaa] for the patch.
v006 LGTM +1
[~elgoiri] anything from your side. :)
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch,
> HDFS-14369-HDFS-13891.005.patch, HDFS-14369-HDFS-13891.006.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14418: Status: Patch Available (was: Open) > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14418-01.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
[ https://issues.apache.org/jira/browse/HDFS-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14418: Attachment: HDFS-14418-01.patch > Remove redundant super user priveledge checks from namenode. > > > Key: HDFS-14418 > URL: https://issues.apache.org/jira/browse/HDFS-14418 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14418-01.patch > > > There are couple of methods that unnecessarily double checks super user > privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14418) Remove redundant super user priveledge checks from namenode.
Ayush Saxena created HDFS-14418: --- Summary: Remove redundant super user priveledge checks from namenode. Key: HDFS-14418 URL: https://issues.apache.org/jira/browse/HDFS-14418 Project: Hadoop HDFS Issue Type: Bug Reporter: Ayush Saxena Assignee: Ayush Saxena There are couple of methods that unnecessarily double checks super user privileged at namenode, which can reduced to single. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812464#comment-16812464
]
Ayush Saxena commented on HDFS-13248:
-
Thanx [~hexiaoqiao] and [~elgoiri]
I guess if we use the {{IpcConnectionContext}} approach, we may even solve the
problem of auditing? and I feel this approach if implemented carefully wouldn't
have any compatibility issues too. Regarding security also I don't see as such
any threat, shall dig in more to verify.
So my personal bet is on this!!!
Regarding client protocol changes, This is like in the present scenario only
router shall be using it and it shall be an optional parameter, but a normal
client shall also be able to use it. We can't stop that being used there so
easily and even this whole shan't be achieved through single change whole read
and write the elementary protocols are the target.
Not against it but, I feel we should make it a second option as of now.There is
no doubt regarding the work-ability of the solution(We are pretty sure this
will work) but would like to offer a chance if we tend to get a better and
safer solution.
Rest, Whichever solution everyone feels good. I shall be happy helping out in
the way in.
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812451#comment-16812451 ] Daryn Sharp commented on HDFS-14406: I'd also prefer this be optional. The overhead will provide little of the intended value if intended for hdfs. Processing time for hdfs ops includes lock wait time which unfairly inflates processing time for all ops. Ie. If one user performs many write operations, all users are billed for the lock wait time while the write ops complete. (Note: this may change soon if my internal patch to exclude lock wait from processing time for the decay scheduler proves successful) Other than that, the following: "One metrics is exposed" should be "A metric is exposed". > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14372) NPE while DN is shutting down
[
https://issues.apache.org/jira/browse/HDFS-14372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812245#comment-16812245
]
lujie commented on HDFS-14372:
--
ping>
@[~ayushtkn] or anyone else, could you please review the new patch?
Thanks!
> NPE while DN is shutting down
> -
>
> Key: HDFS-14372
> URL: https://issues.apache.org/jira/browse/HDFS-14372
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: HDFS-14372_0.patch, HDFS-14372_1.patch,
> HDFS-14372_2.patch
>
>
> Take the code BPServiceActor#register:
> {code:java}
> while (shouldRun()) {
> try {
>// Use returned registration from namenode with updated fields
> newBpRegistration = bpNamenode.registerDatanode(newBpRegistration);
> newBpRegistration.setNamespaceInfo(nsInfo);
> bpRegistration = newBpRegistration;
> break;
> } catch(EOFException e) { // namenode might have just restarted
>
> }
> LOG.info("Block pool " + this + " successfully registered with NN");
> bpos.registrationSucceeded(this, bpRegistration);
> {code}
> if DN is shutdown, then above code will skip the loop, and bpRegistration ==
> null, the null value will be used in DataNode#bpRegistrationSucceeded:
> {code:java}
> if(!storage.getDatanodeUuid().equals(bpRegistration.getDatanodeUuid()))
> {code}
> hence NPE happens
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:1583)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:425)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:807)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:294)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:840)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812173#comment-16812173
]
He Xiaoqiao commented on HDFS-13248:
Thanks [~elgoiri],
{quote}My main concern with modifying ClientProtocol is that it requires the
client itself to change.
The change is backwards compatible but for it to work you need the client to be
up to date.
>From our experience, this is pretty challenging.{quote}
The documents does not depict scope of changing detailed. Actually, we only
need `modify the Namenode and the Router` rather than require change client if
we push to use modifying {{ClientProtocol}}
(1) All client keep to use current interface of {{ClientProtocol}};
(2) When router receive RPC request, it get client hostname firstly, then
switch to invoke additional method which include parameter {{ClientMachine}}
to Namenode;
(3) When RPC request to Namenode, it determine to use {{clientMachine}} if not
null or get client hostname by {{Server.getRemoteAddress}} if {{ClientMachine}}
is null.
In one word, We need to modify Namenode and Router but Client.
{quote}BTW, we could do right away the one that
RouterRpcServer#getBlockLocations() reorders the destinations.{quote}
I agree to reorder at Router again, but I think it's not necessary if we can
handle this case under this ticket. Since reorder operation may reduce QPS of
router. Please correct me if there are something wrong. FYI.
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12245) Update INodeId javadoc
[
https://issues.apache.org/jira/browse/HDFS-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812171#comment-16812171
]
Akira Ajisaka commented on HDFS-12245:
--
How about replacing {{2 << 14 - 1}} with {{1 << 14}}? That seems
straightforward for me.
> Update INodeId javadoc
> --
>
> Key: HDFS-12245
> URL: https://issues.apache.org/jira/browse/HDFS-12245
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Wei-Chiu Chuang
>Assignee: Adam Antal
>Priority: Major
> Attachments: HDFS-12245.001.patch, HDFS-12245.002.patch,
> HDFS-12245.003.patch
>
>
> The INodeId javadoc states that id 1 to 1000 is reserved and root inode id
> start from 1001. That is no longer true after HDFS-4434.
> Also, it's a little weird in INodeId
> {code}
> public static final long LAST_RESERVED_ID = 2 << 14 - 1;
> public static final long ROOT_INODE_ID = LAST_RESERVED_ID + 1;
> {code}
> It seems the intent was for LAST_RESERVED_ID to be (2^14) - 1 = 32767. But
> due to Java operator precedence, LAST_RESERVED_ID = 2^(14-1) = 16384. Maybe
> it doesn't matter, not sure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14303) chek block directory logic not correct when there is only meta file, print no meaning warn log
[ https://issues.apache.org/jira/browse/HDFS-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812151#comment-16812151 ] qiang Liu commented on HDFS-14303: -- Am I submit the patch in wront format, can any one help me how to gernerate a patch? hbase has a script dev-support/submit-patch.py witch can gernerate and subbmit patch, does hdfs has one too? As a very fresh man submiting patch, if i do any thing wrong, please don't mind. > chek block directory logic not correct when there is only meta file, print no > meaning warn log > -- > > Key: HDFS-14303 > URL: https://issues.apache.org/jira/browse/HDFS-14303 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.3 > Environment: env free >Reporter: qiang Liu >Priority: Minor > Labels: easy-fix > Attachments: HDFS-14303-branch-2.7.001.patch, > HDFS-14303-branch-2.7.001.patch, HDFS-14303-branch-2.7.3.002.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > chek block directory logic not correct when there is only meta file,print no > meaning warn log, eg: > WARN DirectoryScanner:? - Block: 1101939874 has to be upgraded to block > ID-based layout. Actual block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68, > expected block file path: > /data14/hadoop/data/current/BP-1461038173-10.8.48.152-1481686842620/current/finalized/subdir174/subdir68/subdir68 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
