[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188066#comment-17188066 ] Íñigo Goiri commented on HDFS-15543: I agree on the idea, I have the feeling it could use some refactoring. What about a function that checks if the exception should be thrown and throw then? > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15545) (S)Webhdfs will not use updated delegation tokens available in the ugi after the old ones expire
[ https://issues.apache.org/jira/browse/HDFS-15545?focusedWorklogId=476839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476839 ] ASF GitHub Bot logged work on HDFS-15545: - Author: ASF GitHub Bot Created on: 31/Aug/20 21:11 Start Date: 31/Aug/20 21:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2255: URL: https://github.com/apache/hadoop/pull/2255#issuecomment-684042566 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 29m 17s | trunk passed | | +1 :green_heart: | compile | 4m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 4m 6s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 1m 24s | trunk passed | | +1 :green_heart: | mvnsite | 2m 58s | trunk passed | | +1 :green_heart: | shadedclient | 20m 33s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 29s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 57s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 2m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 2s | the patch passed | | +1 :green_heart: | compile | 4m 17s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javac | 4m 17s | the patch passed | | +1 :green_heart: | compile | 3m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | javac | 3m 50s | the patch passed | | +1 :green_heart: | checkstyle | 0m 53s | the patch passed | | +1 :green_heart: | mvnsite | 2m 1s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 48s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 1m 19s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 1m 48s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | findbugs | 5m 44s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 59s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 108m 53s | hadoop-hdfs in the patch passed. | | -1 :x: | asflicense | 0m 37s | The patch generated 3 ASF License warnings. | | | | 225m 49s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc | | | hadoop.hdfs.TestFileChecksum | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.server.namenode.ha.TestHAAppend | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2255/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2255 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 03865f2351d8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 4ffe26f9b81 | | Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | unit |
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[
https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187994#comment-17187994
]
Hadoop QA commented on HDFS-15543:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 48m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
19s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
33s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
39s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
|
[jira] [Updated] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15543: --- Status: Patch Available (was: Open) > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15545) (S)Webhdfs will not use updated delegation tokens available in the ugi after the old ones expire
[ https://issues.apache.org/jira/browse/HDFS-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Issac Buenrostro updated HDFS-15545: Attachment: HDFS-15545.002.patch > (S)Webhdfs will not use updated delegation tokens available in the ugi after > the old ones expire > > > Key: HDFS-15545 > URL: https://issues.apache.org/jira/browse/HDFS-15545 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Issac Buenrostro >Assignee: Issac Buenrostro >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15545.001.patch, HDFS-15545.002.patch > > Time Spent: 50m > Remaining Estimate: 0h > > WebHdfsFileSystem can select a delegation token to use from the current user > UGI. The token selection is sticky, and WebHdfsFileSystem will re-use it > every time without searching the UGI again. > If the previous token expires, WebHdfsFileSystem will catch the exception and > attempt to get a new token. However, the mechanism to get a new token > bypasses searching for one on the UGI, so even if there is external logic > that has retrieved a new token, it is not possible to make the FileSystem use > the new, valid token, rendering the FileSystem object unusable. > A simple fix would allow WebHdfsFileSystem to re-search the UGI, and if it > finds a different token than the cached one try to use it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15484) Add new method batchRename for DistributedFileSystem and WebHdfsFileSystem
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15484:
--
Labels: pull-request-available (was: )
> Add new method batchRename for DistributedFileSystem and WebHdfsFileSystem
> --
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Affects Versions: 3.3.0
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-15484.001.patch, HDFS-15484.002.patch,
> HDFS-15484.003.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need one RPC and one lock. For
> example,
> rename(new Path("/dir1/f1::/dir2/f2"), new Path("/dir3/f1::dir4/f4"),
> Rename.BATCH)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15484) Add new method batchRename for DistributedFileSystem and WebHdfsFileSystem
[ https://issues.apache.org/jira/browse/HDFS-15484?focusedWorklogId=476628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476628 ] ASF GitHub Bot logged work on HDFS-15484: - Author: ASF GitHub Bot Created on: 31/Aug/20 15:30 Start Date: 31/Aug/20 15:30 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #2235: URL: https://github.com/apache/hadoop/pull/2235#issuecomment-683852266 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 28m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | buf | 0m 1s | buf was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 25m 58s | trunk passed | | +1 :green_heart: | compile | 19m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | compile | 17m 0s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +1 :green_heart: | checkstyle | 3m 0s | trunk passed | | +1 :green_heart: | mvnsite | 4m 58s | trunk passed | | +1 :green_heart: | shadedclient | 22m 43s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 3m 26s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 | | +1 :green_heart: | javadoc | 5m 10s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 | | +0 :ok: | spotbugs | 1m 24s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 9m 33s | trunk passed | | -0 :warning: | patch | 1m 46s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 22s | hadoop-hdfs-rbf in the patch failed. | | -1 :x: | compile | 5m 25s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | cc | 5m 25s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | javac | 5m 25s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. | | -1 :x: | compile | 4m 53s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | cc | 4m 53s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -1 :x: | javac | 4m 53s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. | | -0 :warning: | checkstyle | 2m 41s | root: The patch generated 13 new + 672 unchanged - 4 fixed = 685 total (was 676) | | -1 :x: | mvnsite | 0m 25s | hadoop-hdfs-rbf in the patch failed. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 42s | patch has no errors when building and testing our client artifacts. | | -1 :x: | javadoc | 0m 36s | hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | -1 :x: | javadoc | 1m 19s | hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1) | | -1 :x: | findbugs | 0m 27s | hadoop-hdfs-rbf in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 9m 6s | hadoop-common in the patch passed. | | +1 :green_heart: | unit | 2m 5s | hadoop-hdfs-client in the patch passed. | | -1 :x: | unit | 108m 7s | hadoop-hdfs in the patch passed. | | -1 :x: | unit | 0m 36s | hadoop-hdfs-rbf in the patch failed. | | -1 :x: | asflicense | 0m 45s | The patch generated 3 ASF License warnings. | | | | 309m 16s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.fs.TestFilterFileSystem | | | hadoop.fs.TestHarFileSystem | | |
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[ https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187783#comment-17187783 ] GANG LIAO commented on HDFS-13443: -- If the system decides to repartition part of the mount table as the result of adding a new NameNode, there will be a period of time where clients may have requests routed to the wrong place. > RBF: Update mount table cache immediately after changing (add/update/remove) > mount table entries. > - > > Key: HDFS-13443 > URL: https://issues.apache.org/jira/browse/HDFS-13443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Mohammad Arshad >Assignee: Mohammad Arshad >Priority: Major > Labels: RBF > Fix For: 3.3.0, HDFS-13891 > > Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch, > HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch, > HDFS-13443-017.patch, HDFS-13443-HDFS-13891-001.patch, > HDFS-13443-HDFS-13891-002.patch, HDFS-13443-branch-2.001.patch, > HDFS-13443-branch-2.002.patch, HDFS-13443.001.patch, HDFS-13443.002.patch, > HDFS-13443.003.patch, HDFS-13443.004.patch, HDFS-13443.005.patch, > HDFS-13443.006.patch, HDFS-13443.007.patch, HDFS-13443.008.patch, > HDFS-13443.009.patch, HDFS-13443.010.patch, HDFS-13443.011.patch > > > Currently mount table cache is updated periodically, by default cache is > updated every minute. After change in mount table, user operations may still > use old mount table. This is bit wrong. > To update mount table cache, maybe we can do following > * *Add refresh API in MountTableManager which will update mount table cache.* > * *When there is a change in mount table entries, router admin server can > update its cache and ask other routers to update their cache*. For example if > there are three routers R1,R2,R3 in a cluster then add mount table entry API, > at admin server side, will perform following sequence of action > ## user submit add mount table entry request on R1 > ## R1 adds the mount table entry in state store > ## R1 call refresh API on R2 > ## R1 calls refresh API on R3 > ## R1 directly freshest its cache > ## Add mount table entry response send back to user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15543: -- Attachment: HDFS-15543.001.patch > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15550) Remove unused imports from TestFileTruncate.java
[
https://issues.apache.org/jira/browse/HDFS-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15550:
---
Issue Type: Improvement (was: Bug)
> Remove unused imports from TestFileTruncate.java
>
>
> Key: HDFS-15550
> URL: https://issues.apache.org/jira/browse/HDFS-15550
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HDFS-15550.001.patch
>
>
> {{import org.apache.hadoop.fs.BlockLocation and import org.junit.Assert
> remain unused in }}{{TestFileTruncate.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15550) Remove unused imports from TestFileTruncate.java
[
https://issues.apache.org/jira/browse/HDFS-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15550:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~Sushma_28] for reports and fix.
Thanks [~brahmareddy] for reviews.
> Remove unused imports from TestFileTruncate.java
>
>
> Key: HDFS-15550
> URL: https://issues.apache.org/jira/browse/HDFS-15550
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HDFS-15550.001.patch
>
>
> {{import org.apache.hadoop.fs.BlockLocation and import org.junit.Assert
> remain unused in }}{{TestFileTruncate.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15550) Remove unused imports from TestFileTruncate.java
[
https://issues.apache.org/jira/browse/HDFS-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187493#comment-17187493
]
Ravuri Sushma sree commented on HDFS-15550:
---
Thanks for the review [~hexiaoqiao]
> Remove unused imports from TestFileTruncate.java
>
>
> Key: HDFS-15550
> URL: https://issues.apache.org/jira/browse/HDFS-15550
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Minor
> Attachments: HDFS-15550.001.patch
>
>
> {{import org.apache.hadoop.fs.BlockLocation and import org.junit.Assert
> remain unused in }}{{TestFileTruncate.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[ https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187409#comment-17187409 ] junbiao chen edited comment on HDFS-14703 at 8/31/20, 6:19 AM: --- [~hexiaoqiao] I want to do some work on this issue ,could you tell me which version does the patch base on?thanks was (Author: dahaishuantuoba): [~hexiaoqiao] I want to do some work on this issue ,could you which version does the patch base on?thanks > NameNode Fine-Grained Locking via Metadata Partitioning > --- > > Key: HDFS-14703 > URL: https://issues.apache.org/jira/browse/HDFS-14703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Konstantin Shvachko >Priority: Major > Attachments: 001-partitioned-inodeMap-POC.tar.gz, NameNode > Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf > > > We target to enable fine-grained locking by splitting the in-memory namespace > into multiple partitions each having a separate lock. Intended to improve > performance of NameNode write operations. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15494) TestReplicaCachingGetSpaceUsed #testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows
[ https://issues.apache.org/jira/browse/HDFS-15494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187463#comment-17187463 ] Ravuri Sushma sree commented on HDFS-15494: --- Above reported test failures are not related to this JIRA > TestReplicaCachingGetSpaceUsed #testReplicaCachingGetSpaceUsedByRBWReplica > Fails on Windows > --- > > Key: HDFS-15494 > URL: https://issues.apache.org/jira/browse/HDFS-15494 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Attachments: HDFS-15494.001.patch > > > TestReplicaCachingGetSpaceUsed #testReplicaCachingGetSpaceUsedByRBWReplica > Fails on Windows because when RBW should be renamed to Finalized, windows is > not supporting . > This should be skipped on Windows -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15550) Remove unused imports from TestFileTruncate.java
[
https://issues.apache.org/jira/browse/HDFS-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187453#comment-17187453
]
Xiaoqiao He commented on HDFS-15550:
[^HDFS-15550.001.patch] LGTM. +1 from my side. Will commit to trunk.
> Remove unused imports from TestFileTruncate.java
>
>
> Key: HDFS-15550
> URL: https://issues.apache.org/jira/browse/HDFS-15550
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ravuri Sushma sree
>Assignee: Ravuri Sushma sree
>Priority: Minor
> Attachments: HDFS-15550.001.patch
>
>
> {{import org.apache.hadoop.fs.BlockLocation and import org.junit.Assert
> remain unused in }}{{TestFileTruncate.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14703) NameNode Fine-Grained Locking via Metadata Partitioning
[
https://issues.apache.org/jira/browse/HDFS-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187451#comment-17187451
]
Xiaoqiao He commented on HDFS-14703:
cc [~shv].
{quote}I want to do some work on this issue ,could you which version does the
patch base on?thanks{quote}
Thanks involve me here. As I know, only sub-task HDFS-14731 has merge to trunk,
others do not commit yet.
> NameNode Fine-Grained Locking via Metadata Partitioning
> ---
>
> Key: HDFS-14703
> URL: https://issues.apache.org/jira/browse/HDFS-14703
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namenode
>Reporter: Konstantin Shvachko
>Priority: Major
> Attachments: 001-partitioned-inodeMap-POC.tar.gz, NameNode
> Fine-Grained Locking.pdf, NameNode Fine-Grained Locking.pdf
>
>
> We target to enable fine-grained locking by splitting the in-memory namespace
> into multiple partitions each having a separate lock. Intended to improve
> performance of NameNode write operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15539) When disallowing snapshot on a dir, throw exception if its trash root is not empty
[
https://issues.apache.org/jira/browse/HDFS-15539?focusedWorklogId=476340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476340
]
ASF GitHub Bot logged work on HDFS-15539:
-
Author: ASF GitHub Bot
Created on: 31/Aug/20 06:07
Start Date: 31/Aug/20 06:07
Worklog Time Spent: 10m
Work Description: bshashikant commented on pull request #2258:
URL: https://github.com/apache/hadoop/pull/2258#issuecomment-683578915
Thanks @smengcl for working on this,. The test failures like
TestDistributedFileSystem#testGetTrashRoots look related. Can you plz verify?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 476340)
Time Spent: 40m (was: 0.5h)
> When disallowing snapshot on a dir, throw exception if its trash root is not
> empty
> --
>
> Key: HDFS-15539
> URL: https://issues.apache.org/jira/browse/HDFS-15539
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When snapshot is disallowed on a dir, {{getTrashRoots()}} won't return the
> trash root in that dir anymore (if any). The risk is the trash root will be
> left there forever.
> We need to throw an exception there and prompt the user to clean up or rename
> the trash root if it is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15542) Add identified snapshot corruption tests for ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDFS-15542. Fix Version/s: 3.4.0 Resolution: Fixed > Add identified snapshot corruption tests for ordered snapshot deletion > -- > > Key: HDFS-15542 > URL: https://issues.apache.org/jira/browse/HDFS-15542 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > HDFS-13101, HDFS-15012 and HDFS-15313 along with HDFS-15470 have fsimage > corruption sequences with snapshots . The idea here is to aggregate these > unit tests and enabled them for ordered snapshot deletion feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15542) Add identified snapshot corruption tests for ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15542?focusedWorklogId=476339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476339 ] ASF GitHub Bot logged work on HDFS-15542: - Author: ASF GitHub Bot Created on: 31/Aug/20 06:00 Start Date: 31/Aug/20 06:00 Worklog Time Spent: 10m Work Description: bshashikant commented on pull request #2251: URL: https://github.com/apache/hadoop/pull/2251#issuecomment-683572317 Thanks @szetszwo for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 476339) Time Spent: 50m (was: 40m) > Add identified snapshot corruption tests for ordered snapshot deletion > -- > > Key: HDFS-15542 > URL: https://issues.apache.org/jira/browse/HDFS-15542 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > HDFS-13101, HDFS-15012 and HDFS-15313 along with HDFS-15470 have fsimage > corruption sequences with snapshots . The idea here is to aggregate these > unit tests and enabled them for ordered snapshot deletion feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15542) Add identified snapshot corruption tests for ordered snapshot deletion
[ https://issues.apache.org/jira/browse/HDFS-15542?focusedWorklogId=476338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476338 ] ASF GitHub Bot logged work on HDFS-15542: - Author: ASF GitHub Bot Created on: 31/Aug/20 06:00 Start Date: 31/Aug/20 06:00 Worklog Time Spent: 10m Work Description: bshashikant merged pull request #2251: URL: https://github.com/apache/hadoop/pull/2251 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 476338) Time Spent: 40m (was: 0.5h) > Add identified snapshot corruption tests for ordered snapshot deletion > -- > > Key: HDFS-15542 > URL: https://issues.apache.org/jira/browse/HDFS-15542 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HDFS-13101, HDFS-15012 and HDFS-15313 along with HDFS-15470 have fsimage > corruption sequences with snapshots . The idea here is to aggregate these > unit tests and enabled them for ordered snapshot deletion feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
