[jira] [Created] (HDFS-15491) Journal transactions batched in sync metrics are missing from Journal Node JMX
Mohit Saindane created HDFS-15491:
-
Summary: Journal transactions batched in sync metrics are missing
from Journal Node JMX
Key: HDFS-15491
URL: https://issues.apache.org/jira/browse/HDFS-15491
Project: Hadoop HDFS
Issue Type: Bug
Components: 3.1.1, hdfs, journal-node, qjm
Affects Versions: 3.1.1
Reporter: Mohit Saindane
Fix For: 3.1.1
Attachments: JournalNode metrics issue.png
I have enable the HA in HDP3.1.5 multinode cluster, I was trying to access the
REST response given by Journal Node on http 8480 port, but as per [Apache
Hadoop 3.1.1
Documentation|[https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/Metrics.html#JournalNode]]
I found that some of the Journal Node specific metrics are missing. (metrics
like NumTransactionsBatchedInSync_(60,300,3600)_sNumOps and
NumTransactionsBatchedInSync_Num_s_(50/75/90/95/99)_thPercentileLatencyMicros),
So checked the _JournalMetrics.java_ code given in
_hadoop-hdfs-3.1.1.3.1.5.0-152-sources.jar_ as well as __ [Hadoop Open Source
Code|[https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalMetrics.java]]
found that the following code is missing :
@Metric("Journal transactions batched in sync")
final MutableQuantiles[] numTransactionsBatchedInSync;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
[
https://issues.apache.org/jira/browse/HDFS-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163228#comment-17163228
]
Hadoop QA commented on HDFS-15490:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 24m
24s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15490 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008213/HDFS-15490.000.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29545/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Address checkstyle issues reported with HDFS-15480
> --
>
> Key: HDFS-15490
> URL: https://issues.apache.org/jira/browse/HDFS-15490
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15490.000.patch
>
>
> {code:java}
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
> class FSDirXAttrOp {:1: Utility classes should not have a public or default
> constructor. [HideUtilityClassConstructor]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
> static final String xattrName = "user.a1";:23: Name 'xattrName' must match
> pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
> static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
> must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15486) Costly sendResponse operation slows down async editlog handling
[
https://issues.apache.org/jira/browse/HDFS-15486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163224#comment-17163224
]
Yuxuan Wang commented on HDFS-15486:
{code:java|title=FSEditLogAsync.java}
public void run() {
...
while ((edit = syncWaitQ.poll()) != null) {
// We should parallel next code ?
edit.logSyncNotify(syncEx);
}
...
}
{code}
Expect your patch.
If I'm wrong, correct me.
> Costly sendResponse operation slows down async editlog handling
> ---
>
> Key: HDFS-15486
> URL: https://issues.apache.org/jira/browse/HDFS-15486
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Yiqun Lin
>Priority: Major
> Attachments: Async-profile-(2).jpg, async-profile-(1).jpg
>
>
> When our cluster NameNode in a very high load, we find it often stuck in
> Async-editlog handling.
> We use async-profile tool to get the flamegraph.
> !Async-profile-(2).jpg!
> This happened in that async editlog thread consumes Edit from the queue and
> triggers the sendResponse call.
> But here the sendResponse call is a little expensive since our cluster
> enabled the security env and will do some encode operations when doing the
> return response operation.
> We often catch some moments of costly sendResponse operation when rpc call
> queue is fulled.
> !async-profile-(1).jpg!
> Slowness on consuming Edit in async editlog will make Edit pending Queue
> easily become the fulled state, then block its enqueue operation that is
> invoked in writeLock type methods in FSNamesystem class.
> Here the enhancement is that we can use multiple thread to parallel execute
> sendResponse call. sendResponse doesn't need use the write lock to do
> protection, so this change is safe.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
[
https://issues.apache.org/jira/browse/HDFS-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15490:
---
Attachment: HDFS-15490.000.patch
> Address checkstyle issues reported with HDFS-15480
> --
>
> Key: HDFS-15490
> URL: https://issues.apache.org/jira/browse/HDFS-15490
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15490.000.patch
>
>
> {code:java}
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
> class FSDirXAttrOp {:1: Utility classes should not have a public or default
> constructor. [HideUtilityClassConstructor]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
> static final String xattrName = "user.a1";:23: Name 'xattrName' must match
> pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
> static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
> must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
[
https://issues.apache.org/jira/browse/HDFS-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDFS-15490:
--
Assignee: Shashikant Banerjee
> Address checkstyle issues reported with HDFS-15480
> --
>
> Key: HDFS-15490
> URL: https://issues.apache.org/jira/browse/HDFS-15490
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15490.000.patch
>
>
> {code:java}
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
> class FSDirXAttrOp {:1: Utility classes should not have a public or default
> constructor. [HideUtilityClassConstructor]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
> static final String xattrName = "user.a1";:23: Name 'xattrName' must match
> pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
> static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
> must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
[
https://issues.apache.org/jira/browse/HDFS-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15490:
---
Status: Patch Available (was: Open)
> Address checkstyle issues reported with HDFS-15480
> --
>
> Key: HDFS-15490
> URL: https://issues.apache.org/jira/browse/HDFS-15490
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15490.000.patch
>
>
> {code:java}
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
> class FSDirXAttrOp {:1: Utility classes should not have a public or default
> constructor. [HideUtilityClassConstructor]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
> static final String xattrName = "user.a1";:23: Name 'xattrName' must match
> pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
> static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
> must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
[
https://issues.apache.org/jira/browse/HDFS-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-15490:
---
Description:
{code:java}
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
class FSDirXAttrOp {:1: Utility classes should not have a public or default
constructor. [HideUtilityClassConstructor]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
static final String xattrName = "user.a1";:23: Name 'xattrName' must match
pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
{code}
> Address checkstyle issues reported with HDFS-15480
> --
>
> Key: HDFS-15490
> URL: https://issues.apache.org/jira/browse/HDFS-15490
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Shashikant Banerjee
>Priority: Major
>
> {code:java}
> ./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java:50:public
> class FSDirXAttrOp {:1: Utility classes should not have a public or default
> constructor. [HideUtilityClassConstructor]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:49:
> static final String xattrName = "user.a1";:23: Name 'xattrName' must match
> pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java:50:
> static final byte[] xattrValue = {0x31, 0x32, 0x33};:23: Name 'xattrValue'
> must match pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'. [ConstantName]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15490) Address checkstyle issues reported with HDFS-15480
Shashikant Banerjee created HDFS-15490: -- Summary: Address checkstyle issues reported with HDFS-15480 Key: HDFS-15490 URL: https://issues.apache.org/jira/browse/HDFS-15490 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Shashikant Banerjee -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163190#comment-17163190
]
Yang Yun commented on HDFS-15484:
-
Thanks [~ste...@apache.org] for your detailed explanation.
Got it, chenges of FS APIs should be more cautious, and also need relevant FS
spec updated.
I leave this jira for disscution. I can continue it if your think it's
worthiness.
The motivation for this batch operation is for the special user. for example,
in the end of spark job, it will rename or delete many files, IPC peak will
impact the perfermance of Namenode. The batch operation is differently
original operation, but it is really can improve the perfermance in our
distribute filesystem.
Yes, we need define the *batch*. Any operations may be in batch, need to
define the ordering, the exception and the return value. one simple way is
client send the operations in fixed order, the server side executes they one by
one. every single operation is same as before. if any one breaks, server side
will immediately return the failed index and failed reason to the client. for
example,
rename(/a:/b:/c, /b:/c:/a)
step 1: (/a > /b) success
step 2 : (/b > /c) success
step 3: (/c > /a) success
return 3, Actually, do nothing.
rename(/a:/a/subdir, /b/c:/b)
step 1: ( /a > /b/c ) success
step 2: (/a/subdir, /b) failed for "/a/subdir" is not existing.
will return the faile number 1 and the failed reason ("/a/subdir" is not
existent)
rename(/a:/nonexistent, /b:/c)
step 1: (/a > /b) success
step 2: (/nonexistent > /c) failed
will return number 1 and failed reason (/nonexistent is not existent)
rename(/a:/b, /c:/c)
step 1: (/a > /c) success
step 2: (/b > /c) failed
will return 1 and half failed reason (/c is existent)
Anyway, the single operation is same as before. only run one by one on the
server side.
the server will return a exception if it's not support batch. other filesystem
can throw exception for the batch.
We can combine paths in other ways, for example, json format.
I also think async opertaion is very valuable, especially for batch operations.
If something needs me to do, please let me know.
> Add option in enum Rename to suport batch rename
>
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15484.001.patch
>
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need one RPC and one lock. For
> example,
> rename(new Path("/dir1/f1::/dir2/f2"), new Path("/dir3/f1::dir4/f4"),
> Rename.BATCH)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15489) Documentation link is broken for Apache Hadoop
[ https://issues.apache.org/jira/browse/HDFS-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Maheshwari updated HDFS-15489: Attachment: Screen Shot 2020-07-22 at 5.24.29 PM.png > Documentation link is broken for Apache Hadoop > -- > > Key: HDFS-15489 > URL: https://issues.apache.org/jira/browse/HDFS-15489 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Namit Maheshwari >Priority: Major > Attachments: DocumentLinkBroken.mov, Screen Shot 2020-07-22 at > 5.24.22 PM.png, Screen Shot 2020-07-22 at 5.24.29 PM.png > > > > Please see attached video and screenshots > [^DocumentLinkBroken.mov] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15489) Documentation link is broken for Apache Hadoop
[ https://issues.apache.org/jira/browse/HDFS-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Maheshwari updated HDFS-15489: Attachment: Screen Shot 2020-07-22 at 5.24.22 PM.png > Documentation link is broken for Apache Hadoop > -- > > Key: HDFS-15489 > URL: https://issues.apache.org/jira/browse/HDFS-15489 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: Namit Maheshwari >Priority: Major > Attachments: DocumentLinkBroken.mov, Screen Shot 2020-07-22 at > 5.24.22 PM.png, Screen Shot 2020-07-22 at 5.24.29 PM.png > > > > Please see attached video and screenshots > [^DocumentLinkBroken.mov] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15489) Documentation link is broken for Apache Hadoop
Namit Maheshwari created HDFS-15489: --- Summary: Documentation link is broken for Apache Hadoop Key: HDFS-15489 URL: https://issues.apache.org/jira/browse/HDFS-15489 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Namit Maheshwari Attachments: DocumentLinkBroken.mov Please see attached video and screenshots [^DocumentLinkBroken.mov] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[ https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163139#comment-17163139 ] Hudson commented on HDFS-15480: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18466 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18466/]) HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (github: rev 2d12496643b1b7cfa4eb270ec9b2fcdb78a58798) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestOrderedSnapshotDeletion.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java > Ordered snapshot deletion: record snapshot deletion in XAttr > > > Key: HDFS-15480 > URL: https://issues.apache.org/jira/browse/HDFS-15480 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 1.3.0 > > Attachments: HDFS-15480.000.patch, HDFS-15480.001.patch, > HDFS-15480.002.patch > > > In this JIRA, the behavior of deleting the non-earliest snapshots will be > changed to marking them as deleted in XAttr but not actually deleting them. > Note that > # The marked-for-deletion snapshots will be garbage collected later on; see > HDFS-15481. > # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[ https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze updated HDFS-15480: -- Fix Version/s: 1.3.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) I have merged the pull request. Thanks, [~shashikant]! > Ordered snapshot deletion: record snapshot deletion in XAttr > > > Key: HDFS-15480 > URL: https://issues.apache.org/jira/browse/HDFS-15480 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 1.3.0 > > Attachments: HDFS-15480.000.patch, HDFS-15480.001.patch, > HDFS-15480.002.patch > > > In this JIRA, the behavior of deleting the non-earliest snapshots will be > changed to marking them as deleted in XAttr but not actually deleting them. > Note that > # The marked-for-deletion snapshots will be garbage collected later on; see > HDFS-15481. > # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[ https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163130#comment-17163130 ] Tsz-wo Sze commented on HDFS-15480: --- +1 the 002 patch looks good. > Ordered snapshot deletion: record snapshot deletion in XAttr > > > Key: HDFS-15480 > URL: https://issues.apache.org/jira/browse/HDFS-15480 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-15480.000.patch, HDFS-15480.001.patch, > HDFS-15480.002.patch > > > In this JIRA, the behavior of deleting the non-earliest snapshots will be > changed to marking them as deleted in XAttr but not actually deleting them. > Note that > # The marked-for-deletion snapshots will be garbage collected later on; see > HDFS-15481. > # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15482) Ordered snapshot deletion: hide the deleted snapshots from users
[ https://issues.apache.org/jira/browse/HDFS-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163128#comment-17163128 ] Tsz-wo Sze commented on HDFS-15482: --- Hi [~hemanthboyina], the marked-as-deleted snapshots will be hided from users for all the operations including list, get and rename. Users will get FileNotFoundException (or a similar exception) as if the snapshot is indeed deleted. > Ordered snapshot deletion: hide the deleted snapshots from users > > > Key: HDFS-15482 > URL: https://issues.apache.org/jira/browse/HDFS-15482 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Shashikant Banerjee >Priority: Major > > In HDFS-15480, the behavior of deleting the non-earliest snapshots is > changed to marking them as deleted in XAttr but not actually deleting them. > The users are still able to access the these snapshots as usual. > In this JIRA, the marked-for-deletion snapshots are hided so that they become > inaccessible > to users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12969) DfsAdmin listOpenFiles should report files by type
[
https://issues.apache.org/jira/browse/HDFS-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163127#comment-17163127
]
Hadoop QA commented on HDFS-12969:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
39s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 21m 45s{color} |
{color:red} root generated 28 new + 134 unchanged - 28 fixed = 162 total (was
162) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 50s{color}
| {color:red} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
31s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 42s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}293m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestFixKerberosTicketOrder |
| |
[jira] [Commented] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[
https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163113#comment-17163113
]
Hadoop QA commented on HDFS-15480:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
38s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 98 unchanged - 1 fixed = 101 total (was 99) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 22s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29544/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008196/HDFS-15480.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 94f69956fd9b
[jira] [Commented] (HDFS-15473) Add listSnapshots command to list all snapshots
[
https://issues.apache.org/jira/browse/HDFS-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163089#comment-17163089
]
Hadoop QA commented on HDFS-15473:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
18s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 17s{color}
| {color:red} hadoop-hdfs-project generated 1 new + 753 unchanged - 0 fixed =
754 total (was 753) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-hdfs-project: The patch generated 8 new +
502 unchanged - 0 fixed = 510 total (was 502) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
34s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
14s{color} | {color:green} The patch generated 0 new + 104 unchanged - 132
fixed = 104 total (was 236) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 3 new
+ 97 unchanged - 3 fixed = 100 total (was 100) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 7s{color}
| {color:red} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 27s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
|
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[
https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163078#comment-17163078
]
Hadoop QA commented on HDFS-15098:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 22m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
39m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 33m
12s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 40m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 8s{color} |
{color:red} root generated 44 new + 118 unchanged - 44 fixed = 162 total (was
162) {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 17m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 8s{color}
| {color:red} root generated 4 new + 1948 unchanged - 0 fixed = 1952 total (was
1948) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 40s{color} | {color:orange} root: The patch generated 3 new + 227 unchanged
- 8 fixed = 230 total (was 235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 38m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}582m 0s{color}
| {color:red} root in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
36s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
[jira] [Updated] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[ https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15480: --- Attachment: HDFS-15480.002.patch > Ordered snapshot deletion: record snapshot deletion in XAttr > > > Key: HDFS-15480 > URL: https://issues.apache.org/jira/browse/HDFS-15480 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-15480.000.patch, HDFS-15480.001.patch, > HDFS-15480.002.patch > > > In this JIRA, the behavior of deleting the non-earliest snapshots will be > changed to marking them as deleted in XAttr but not actually deleting them. > Note that > # The marked-for-deletion snapshots will be garbage collected later on; see > HDFS-15481. > # The marked-for-deletion snapshots will be hided from users; see HDFS-15482. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12969) DfsAdmin listOpenFiles should report files by type
[
https://issues.apache.org/jira/browse/HDFS-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162986#comment-17162986
]
Hemanth Boyina commented on HDFS-12969:
---
thanks [~tasanuma] for the review
{quote}TestDecommission#testListOpenFilesWithType failed when
{{dfs.namenode.list.openfiles.num.responses}} is not 1 (like 2 or 3). Some
BLOCKING_DECOMMISSION files are incorrectly listed as ALL_OPEN_FILES
{quote}
updated the patch by adding a new test case for different number of responses
{quote} I think NameNode needs to have the list of ALL_OPEN_FILES(non-blocking)
files in advance
{quote}
agree that , having this will make the improvement easier , but i think
Namenode having all openfiles without blocking decommission in advance will
not be benefited/useful in our current scenarios except listOpenFilesByType
> DfsAdmin listOpenFiles should report files by type
> --
>
> Key: HDFS-12969
> URL: https://issues.apache.org/jira/browse/HDFS-12969
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Manoj Govindassamy
>Assignee: Hemanth Boyina
>Priority: Major
> Attachments: HDFS-12969.001.patch, HDFS-12969.002.patch,
> HDFS-12969.003.patch
>
>
> HDFS-11847 has introduced a new option to {{-blockingDecommission}} to an
> existing command
> {{dfsadmin -listOpenFiles}}. But the reporting done by the command doesn't
> differentiate the files based on the type (like blocking decommission). In
> order to change the reporting style, the proto format used for the base
> command has to be updated to carry additional fields and better be done in a
> new jira outside of HDFS-11847. This jira is to track the end-to-end
> enhancements needed for dfsadmin -listOpenFiles console output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12969) DfsAdmin listOpenFiles should report files by type
[
https://issues.apache.org/jira/browse/HDFS-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina updated HDFS-12969:
--
Attachment: HDFS-12969.003.patch
> DfsAdmin listOpenFiles should report files by type
> --
>
> Key: HDFS-12969
> URL: https://issues.apache.org/jira/browse/HDFS-12969
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.0
>Reporter: Manoj Govindassamy
>Assignee: Hemanth Boyina
>Priority: Major
> Attachments: HDFS-12969.001.patch, HDFS-12969.002.patch,
> HDFS-12969.003.patch
>
>
> HDFS-11847 has introduced a new option to {{-blockingDecommission}} to an
> existing command
> {{dfsadmin -listOpenFiles}}. But the reporting done by the command doesn't
> differentiate the files based on the type (like blocking decommission). In
> order to change the reporting style, the proto format used for the base
> command has to be updated to carry additional fields and better be done in a
> new jira outside of HDFS-11847. This jira is to track the end-to-end
> enhancements needed for dfsadmin -listOpenFiles console output.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15473) Add listSnapshots command to list all snapshots
[ https://issues.apache.org/jira/browse/HDFS-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15473: -- Attachment: HDFS-15473.001.patch Status: Patch Available (was: Open) > Add listSnapshots command to list all snapshots > --- > > Key: HDFS-15473 > URL: https://issues.apache.org/jira/browse/HDFS-15473 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15473.001.patch > > > In a cluster where snapshots are highly used , it will benefit to have a > command to list all the snapshots under root -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15443) Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure.
[
https://issues.apache.org/jira/browse/HDFS-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162946#comment-17162946
]
Íñigo Goiri commented on HDFS-15443:
Some of the failed tests seem pretty consistent.
Can we make sure they are not broken?
> Setting dfs.datanode.max.transfer.threads to a very small value can cause
> strange failure.
> --
>
> Key: HDFS-15443
> URL: https://issues.apache.org/jira/browse/HDFS-15443
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: AMC-team
>Priority: Major
> Attachments: HDFS-15443.000.patch, HDFS-15443.001.patch,
> HDFS-15443.002.patch
>
>
> Configuration parameter dfs.datanode.max.transfer.threads is to specify the
> maximum number of threads to use for transferring data in and out of the DN.
> This is a vital param that need to tune carefully.
> {code:java}
> // DataXceiverServer.java
> // Make sure the xceiver count is not exceeded
> intcurXceiverCount = datanode.getXceiverCount();
> if (curXceiverCount > maxXceiverCount) {
> thrownewIOException("Xceiver count " + curXceiverCount
> + " exceeds the limit of concurrent xceivers: "
> + maxXceiverCount);
> }
> {code}
> There are many issues that caused by not setting this param to an appropriate
> value. However, there is no any check code to restrict the parameter.
> Although having a hard-and-fast rule is difficult because we need to consider
> number of cores, main memory etc, *we can prevent users from setting this
> value to an absolute wrong value by accident.* (e.g. a negative value that
> totally break the availability of datanode.)
> *How to fix:*
> Add proper check code for the parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15480) Ordered snapshot deletion: record snapshot deletion in XAttr
[
https://issues.apache.org/jira/browse/HDFS-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162943#comment-17162943
]
Hadoop QA commented on HDFS-15480:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
18s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15480 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008070/HDFS-15480.001.patch |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-2163/6/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Ordered snapshot deletion: record snapshot deletion in XAttr
>
>
> Key: HDFS-15480
> URL: https://issues.apache.org/jira/browse/HDFS-15480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: snapshots
>Reporter: Tsz-wo Sze
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-15480.000.patch, HDFS-15480.001.patch
>
>
> In this JIRA, the behavior of deleting the non-earliest snapshots will be
> changed to marking them as deleted in XAttr but not actually deleting them.
> Note that
> # The marked-for-deletion snapshots will be garbage collected later on; see
> HDFS-15481.
> # The marked-for-deletion snapshots will be hided from users; see HDFS-15482.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162827#comment-17162827
]
Hadoop QA commented on HDFS-15484:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
1s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
6s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 16m 28s{color} |
{color:red} root generated 28 new + 134 unchanged - 28 fixed = 162 total (was
162) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green} root: The patch generated 0 new + 354 unchanged - 1
fixed = 354 total (was 355) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
29s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 13s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}232m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15486) Costly sendResponse operation slows down async editlog handling
[ https://issues.apache.org/jira/browse/HDFS-15486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162780#comment-17162780 ] Yuanbo Liu commented on HDFS-15486: --- I happendly found a similiar issue discribing performance regression while using high version of open-ssl. Here is the link, not sure whether it's helpful. [https://github.com/edenhill/librdkafka/issues/2229] > Costly sendResponse operation slows down async editlog handling > --- > > Key: HDFS-15486 > URL: https://issues.apache.org/jira/browse/HDFS-15486 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Yiqun Lin >Priority: Major > Attachments: Async-profile-(2).jpg, async-profile-(1).jpg > > > When our cluster NameNode in a very high load, we find it often stuck in > Async-editlog handling. > We use async-profile tool to get the flamegraph. > !Async-profile-(2).jpg! > This happened in that async editlog thread consumes Edit from the queue and > triggers the sendResponse call. > But here the sendResponse call is a little expensive since our cluster > enabled the security env and will do some encode operations when doing the > return response operation. > We often catch some moments of costly sendResponse operation when rpc call > queue is fulled. > !async-profile-(1).jpg! > Slowness on consuming Edit in async editlog will make Edit pending Queue > easily become the fulled state, then block its enqueue operation that is > invoked in writeLock type methods in FSNamesystem class. > Here the enhancement is that we can use multiple thread to parallel execute > sendResponse call. sendResponse doesn't need use the write lock to do > protection, so this change is safe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162752#comment-17162752
]
Steve Loughran commented on HDFS-15484:
---
First: As this goes near the hadoop FS APIs, it needs a matching hadoop-common
JIRA for broader visibility, and the relevant FS spec updated, contract tests.
The HDFS team do often forget that expectation, which creates needless tension,
etc (HADOOP-16898)
I can see the benefits of this for HDFS and that this is straightforward to
implement there, especially if you define it's semantics and path element
validity as "what HDFS does"
I worry about object storage though -what if they don't implement it, and if
they do, what exactly are they expected to do.
At a quick glance, there is no immediate benefit in the object stores, and
viewFS won't be able to handle renames across mounts. If any of the object
stores implemented it, without contract tests or specifications is a risk that
implement it "Wrong". And we all know about how much trouble rename() is
already.
Meanwhile, people coding for this need to know how to use it, whether it can
fail halfway through or not -and what that means, and how to identify and cope
with far systems which don't implement it.
If you really do want to do this, you are going to have to make sure the API
works
# define *batch* in this world
# Semantics, including atomicity, partial failures, ordering of renames
# what would rename-(/a:/b:/c, /b:/c:/a:) do?
# rename(/a:/a/subdir, /b/c:/b) ?
# rename(/a:/nonexistent, /b:/c)
# rename(/a:/b, /c:/c are)
# rename dir onto file, file onto file, parent dir doesn't exist, parent is a
file,...
# what does an FS which doesn't support the operation do?
# how current applications discover whether or not this option is supported?
PathCapabilities?
# how do all the current filesystems other than HDFS react to a batch call? (at
a glance: badly)
# what to do on a filesystem with multiple mounts, e.g viewFS
# what about filesystems where ":" is a valid element of a path? recurrent
issue:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HADOOP%20AND%20text%20~%20%22path%20colon%22
# where are the contract tests, especially for the troublespots?
Yes, a lot of these things seem "obvious", but the semantics of rename() make
it the hardest API call to implement, and the fact that HDFS behaves
differently from posix is part of the problem:
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md#boolean-renamepath-src-path-d
FWIW, I've been thinking recently about doing an async rename in the form of a
builder API like openFile; one where clients can block waiting for a response
(one which can even include the statistics of HADOOP-16839). All those stores
with O(files) or O(data) rename semantics would then let the caller do other
things during rename; if we adde,
{code}
Future f = renameFile(sourceFile)
.withDestFile(dest) // must be a filename with a parent path existing, no
file at end
.withDestPermissions(0777)
.build()
f.get()
Future f = renameDir(srcDir)
.withDestDir() // must be a nonexist dir with parent existing
.must("fs.operation.rename.atomic", true) // request atomic dir rename;
stores which don't MUST fail
.build()
Why the separate entries: caller has to know what they are doing. Which
generally they do, we just lose that fact in the API call and end up having to
look at the remote FS to work out what to do.
I've not done any work on this. If we did start it, a batch sequence would be
similar
Future f = renameBatch(parent of all renames) // allows
viewfs to forward
.withDir(src, dest)
.withFile(src, dest)
...
build
+add all the definitons about order etc, ideally with safety checks for obvious
conflicts (same src, same dest, chain) to fail consistently there.
Reporting the state after a partial failure would be tricky though.
If changes to rename are being done, it would be time to do a cloud-native
rename operation. I have no plans to write this, so if someone where to start
the new API for both FS and FC, I'm happy to let them...
(Sorry for causing trouble, but rename is the FS API which causes more problems
in cloud land than any other)
> Add option in enum Rename to suport batch rename
>
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15484.001.patch
>
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need
[jira] [Updated] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Yun updated HDFS-15484:
Status: Open (was: Patch Available)
> Add option in enum Rename to suport batch rename
>
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need one RPC and one lock. For
> example,
> rename(new Path("/dir1/f1::/dir2/f2"), new Path("/dir3/f1::dir4/f4"),
> Rename.BATCH)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Yun updated HDFS-15484:
Attachment: HDFS-15484.001.patch
Status: Patch Available (was: Open)
> Add option in enum Rename to suport batch rename
>
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15484.001.patch
>
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need one RPC and one lock. For
> example,
> rename(new Path("/dir1/f1::/dir2/f2"), new Path("/dir3/f1::dir4/f4"),
> Rename.BATCH)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15484) Add option in enum Rename to suport batch rename
[
https://issues.apache.org/jira/browse/HDFS-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Yun updated HDFS-15484:
Attachment: (was: HDFS-15484.001.patch)
> Add option in enum Rename to suport batch rename
>
>
> Key: HDFS-15484
> URL: https://issues.apache.org/jira/browse/HDFS-15484
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient, namenode, performance
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
>
> Sometime we need rename many files after a task, add a new option in enum
> Rename to support batch rename, which only need one RPC and one lock. For
> example,
> rename(new Path("/dir1/f1::/dir2/f2"), new Path("/dir3/f1::dir4/f4"),
> Rename.BATCH)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[
https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162613#comment-17162613
]
Hadoop QA commented on HDFS-15098:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 24m
21s{color} | {color:red} Docker failed to build yetus/hadoop:cce5a6f6094.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-15098 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13008138/HDFS-15098.009.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29540/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Add SM4 encryption method for HDFS
> --
>
> Key: HDFS-15098
> URL: https://issues.apache.org/jira/browse/HDFS-15098
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 3.4.0
>Reporter: liusheng
>Assignee: zZtai
>Priority: Major
> Labels: sm4
> Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch,
> HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch,
> HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch,
> HDFS-15098.009.patch
>
>
> SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard
> for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure).
> SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far
> been rejected by ISO. One of the reasons for the rejection has been
> opposition to the WAPI fast-track proposal by the IEEE. please see:
> [https://en.wikipedia.org/wiki/SM4_(cipher)]
>
> *Use sm4 on hdfs as follows:*
> 1.Configure Hadoop KMS
> 2.test HDFS sm4
> hadoop key create key1 -cipher 'SM4/CTR/NoPadding'
> hdfs dfs -mkdir /benchmarks
> hdfs crypto -createZone -keyName key1 -path /benchmarks
> *requires:*
> 1.openssl version >=1.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: HDFS-15098.009.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Status: Open (was: Patch Available) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15488) Add a command to list all snapshots for a snaphottable root with snapshot Ids
[ https://issues.apache.org/jira/browse/HDFS-15488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15488: --- Summary: Add a command to list all snapshots for a snaphottable root with snapshot Ids (was: Add. a command to list all snapshots for a snaphottable root with snap Ids) > Add a command to list all snapshots for a snaphottable root with snapshot Ids > - > > Key: HDFS-15488 > URL: https://issues.apache.org/jira/browse/HDFS-15488 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > Currently, the way to list snapshots is do a ls on > /.snapshot directory. Since creation time is not > recorded , there is no way to actually figure out the chronological order of > snapshots. The idea here is to add a command to list snapshots for a > snapshottable directory along with snapshot Ids which grow monotonically as > snapshots are created in the system. With snapID, it will be helpful to > figure out the chronology of snapshots in the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: (was: HDFS-15098.009.patch) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Attachment: HDFS-15098.009.patch Status: Patch Available (was: Open) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zZtai updated HDFS-15098: - Status: Open (was: Patch Available) > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: zZtai >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15488) Add. a command to list all snapshots for a snaphottable root with snap Ids
Shashikant Banerjee created HDFS-15488: -- Summary: Add. a command to list all snapshots for a snaphottable root with snap Ids Key: HDFS-15488 URL: https://issues.apache.org/jira/browse/HDFS-15488 Project: Hadoop HDFS Issue Type: Sub-task Components: snapshots Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Currently, the way to list snapshots is do a ls on /.snapshot directory. Since creation time is not recorded , there is no way to actually figure out the chronological order of snapshots. The idea here is to add a command to list snapshots for a snapshottable directory along with snapshot Ids which grow monotonically as snapshots are created in the system. With snapID, it will be helpful to figure out the chronology of snapshots in the system. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15478) When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.
[ https://issues.apache.org/jira/browse/HDFS-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162532#comment-17162532 ] Hudson commented on HDFS-15478: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18463 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18463/]) HDFS-15478: When Empty mount points, we are assigning fallback link to (github: rev ac9a07b51aefd0fd3b4602adc844ab0f172835e3) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsOverloadSchemeListStatus.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java > When Empty mount points, we are assigning fallback link to self. But it > should not use full URI for target fs. > -- > > Key: HDFS-15478 > URL: https://issues.apache.org/jira/browse/HDFS-15478 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 3.4.0 > > > On empty mount tables detection, we will automatically assign fallback with > the same initialized uri fs. Currently we are using given uri for creating > target fs. > When creating target fs, we use Chrooted fs where it will set the path from > uri as base directory. So, this can make path wrong in the case of fs > initialized with path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15478) When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.
[ https://issues.apache.org/jira/browse/HDFS-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15478: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thank you [~ayushtkn] for review! > When Empty mount points, we are assigning fallback link to self. But it > should not use full URI for target fs. > -- > > Key: HDFS-15478 > URL: https://issues.apache.org/jira/browse/HDFS-15478 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 3.4.0 > > > On empty mount tables detection, we will automatically assign fallback with > the same initialized uri fs. Currently we are using given uri for creating > target fs. > When creating target fs, we use Chrooted fs where it will set the path from > uri as base directory. So, this can make path wrong in the case of fs > initialized with path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
