[jira] [Commented] (HDFS-16971) Add read time metrics for remote reads in Statistics
[ https://issues.apache.org/jira/browse/HDFS-16971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708619#comment-17708619 ] ASF GitHub Bot commented on HDFS-16971: --- melissayou commented on PR #5534: URL: https://github.com/apache/hadoop/pull/5534#issuecomment-1496772546 Hi @goiri , can you help review this PR? Thanks! > Add read time metrics for remote reads in Statistics > > > Key: HDFS-16971 > URL: https://issues.apache.org/jira/browse/HDFS-16971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Melissa You >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.0, 3.4.0 > > > In Filesystem.java, currently it only collects bytes read not the > corresponding timing of reads. > In particular, we're interested in time spent on remote reads(not local > reads). Adding timing info will help us understand it better and further > analyze how locality of DN behaves. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock
[
https://issues.apache.org/jira/browse/HDFS-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708617#comment-17708617
]
ASF GitHub Bot commented on HDFS-16973:
---
hadoop-yetus commented on PR #5533:
URL: https://github.com/apache/hadoop/pull/5533#issuecomment-1496758029
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 1m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 27s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 125m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5533/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5533 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9db22699164b 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a1c34189faab01208d2ed3bbe9c2c1dd1fa5722 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5533/1/testReport/ |
| Max. process+thread count | 2697 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5533/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF:
[jira] [Commented] (HDFS-16971) Add read time metrics for remote reads in Statistics
[ https://issues.apache.org/jira/browse/HDFS-16971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708613#comment-17708613 ] ASF GitHub Bot commented on HDFS-16971: --- melissayou opened a new pull request, #5534: URL: https://github.com/apache/hadoop/pull/5534 ### Description of PR In Filesystem.java, currently it only collects bytes read not the corresponding timing of reads. In particular, we're interested in time spent on remote reads(not local reads). Adding timing info will help us understand it better and further analyze how locality of DN behaves. ### How was this patch tested? Ran hadoop-common and hadoop-hdsf related unit tests locally with and without the change. ### For code changes: - [X ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add read time metrics for remote reads in Statistics > > > Key: HDFS-16971 > URL: https://issues.apache.org/jira/browse/HDFS-16971 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Melissa You >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.0, 3.4.0 > > > In Filesystem.java, currently it only collects bytes read not the > corresponding timing of reads. > In particular, we're interested in time spent on remote reads(not local > reads). Adding timing info will help us understand it better and further > analyze how locality of DN behaves. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock
[ https://issues.apache.org/jira/browse/HDFS-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16973: -- Labels: pull-request-available (was: ) > RBF: MountTableResolver cache size lookup should take read lock > --- > > Key: HDFS-16973 > URL: https://issues.apache.org/jira/browse/HDFS-16973 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Mount table resolver location cache gets invalidated by taking write lock as > part of addEntry/removeEntry/refreshEntries calls. Since the write lock > exclusively updates the cache, getDestinationForPath already takes read lock > before accessing the cache. Similarly, retrieval of the cache size should > also take the read lock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock
[ https://issues.apache.org/jira/browse/HDFS-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708607#comment-17708607 ] ASF GitHub Bot commented on HDFS-16973: --- virajjasani opened a new pull request, #5533: URL: https://github.com/apache/hadoop/pull/5533 Mount table resolver location cache gets invalidated by taking write lock as part of addEntry/removeEntry/refreshEntries calls. Since the write lock exclusively updates the cache, getDestinationForPath already takes read lock before accessing the cache. Similarly, retrieval of the cache size should also take the read lock. > RBF: MountTableResolver cache size lookup should take read lock > --- > > Key: HDFS-16973 > URL: https://issues.apache.org/jira/browse/HDFS-16973 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > Mount table resolver location cache gets invalidated by taking write lock as > part of addEntry/removeEntry/refreshEntries calls. Since the write lock > exclusively updates the cache, getDestinationForPath already takes read lock > before accessing the cache. Similarly, retrieval of the cache size should > also take the read lock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16973) RBF: MountTableResolver cache size lookup should take read lock
Viraj Jasani created HDFS-16973: --- Summary: RBF: MountTableResolver cache size lookup should take read lock Key: HDFS-16973 URL: https://issues.apache.org/jira/browse/HDFS-16973 Project: Hadoop HDFS Issue Type: Improvement Reporter: Viraj Jasani Assignee: Viraj Jasani Mount table resolver location cache gets invalidated by taking write lock as part of addEntry/removeEntry/refreshEntries calls. Since the write lock exclusively updates the cache, getDestinationForPath already takes read lock before accessing the cache. Similarly, retrieval of the cache size should also take the read lock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16945) RBF: add RouterSecurityAuditLogger for router security manager
[
https://issues.apache.org/jira/browse/HDFS-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708586#comment-17708586
]
ASF GitHub Bot commented on HDFS-16945:
---
simbadzina commented on code in PR #5468:
URL: https://github.com/apache/hadoop/pull/5468#discussion_r1157760199
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/RouterSecurityAuditLogger.java:
##
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router.security;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.CallerContext;
+import org.apache.hadoop.ipc.Server;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.InetAddress;
+
+import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.*;
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_SIGNATURE_MAX_SIZE_DEFAULT;
+
+public class RouterSecurityAuditLogger {
+
+ public static final Logger AUDIT_LOG = LoggerFactory.getLogger(
+ RouterSecurityManager.class.getName() + ".audit");
+
+ private static final ThreadLocal STRING_BUILDER =
+ new ThreadLocal() {
+@Override
+protected StringBuilder initialValue() {
+ return new StringBuilder();
+}
+ };
+
+ private int callerContextMaxLen;
+ private int callerSignatureMaxLen;
+
+ public RouterSecurityAuditLogger(Configuration conf) {
+callerContextMaxLen = conf.getInt(
+HADOOP_CALLER_CONTEXT_MAX_SIZE_KEY,
+HADOOP_CALLER_CONTEXT_MAX_SIZE_DEFAULT);
+callerSignatureMaxLen = conf.getInt(
+HADOOP_CALLER_CONTEXT_SIGNATURE_MAX_SIZE_KEY,
+HADOOP_CALLER_CONTEXT_SIGNATURE_MAX_SIZE_DEFAULT);
+ }
+
+ public void logAuditEvent(boolean succeeded, String userName,
+InetAddress addr, String cmd,
+CallerContext callerContext, String tokenId) {
+if (AUDIT_LOG.isDebugEnabled() || AUDIT_LOG.isInfoEnabled()) {
+ logAuditMessage(
+ creatAuditLog(succeeded, userName, addr, cmd, callerContext,
+ tokenId));
+}
+ }
+
+ @VisibleForTesting
+ public String creatAuditLog(boolean succeeded, String userName,
Review Comment:
Typo `createAuditLog`
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/RouterSecurityManager.java:
##
@@ -152,7 +160,8 @@ public Token
getDelegationToken(Text renewer)
tokenId = dtId.toStringStable();
success = true;
} finally {
- logAuditEvent(success, operationName, tokenId);
+ logAuditEvent(success, user, Server.getRemoteIp(), operationName,
Review Comment:
The remote address should be part of the CallerContext as well after
HDFS-13248.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/RouterSecurityAuditLogger.java:
##
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router.security;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import
[jira] [Commented] (HDFS-16943) RBF: Implement MySQL based StateStoreDriver
[ https://issues.apache.org/jira/browse/HDFS-16943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708575#comment-17708575 ] ASF GitHub Bot commented on HDFS-16943: --- simbadzina commented on PR #5469: URL: https://github.com/apache/hadoop/pull/5469#issuecomment-1496573814 Thanks @goiri for the review and approval. Could you please squash and merge the commits when appropriate. > RBF: Implement MySQL based StateStoreDriver > --- > > Key: HDFS-16943 > URL: https://issues.apache.org/jira/browse/HDFS-16943 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs, rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > RBF supports two types of StateStoreDrivers > # StateStoreFileImpl > # StateStoreZooKeeperImpl > I propose implementing a third driver that is backed by MySQL. > * StateStoreZooKeeperImpl requires an additional Zookeeper cluster. > * StateStoreFileImpl can use one of the namenodes in the HDFS cluster, but > that namenode becomes a single point of failure, introducing coupling between > the federated clusters. > HADOOP-18535 implemented a MySQL token store. When tokens are stored in > MySQL, using MySQL for the StateStore as well reduces the number of external > dependencies for routers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16943) RBF: Implement MySQL based StateStoreDriver
[ https://issues.apache.org/jira/browse/HDFS-16943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708559#comment-17708559 ] ASF GitHub Bot commented on HDFS-16943: --- simbadzina commented on PR #5469: URL: https://github.com/apache/hadoop/pull/5469#issuecomment-1496527685 @tasanuma could you please take a look at this pull request when you can. Thanks. > RBF: Implement MySQL based StateStoreDriver > --- > > Key: HDFS-16943 > URL: https://issues.apache.org/jira/browse/HDFS-16943 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs, rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > RBF supports two types of StateStoreDrivers > # StateStoreFileImpl > # StateStoreZooKeeperImpl > I propose implementing a third driver that is backed by MySQL. > * StateStoreZooKeeperImpl requires an additional Zookeeper cluster. > * StateStoreFileImpl can use one of the namenodes in the HDFS cluster, but > that namenode becomes a single point of failure, introducing coupling between > the federated clusters. > HADOOP-18535 implemented a MySQL token store. When tokens are stored in > MySQL, using MySQL for the StateStore as well reduces the number of external > dependencies for routers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16952) Support getLinkTarget API in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-16952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708476#comment-17708476
]
ASF GitHub Bot commented on HDFS-16952:
---
hadoop-yetus commented on PR #5517:
URL: https://github.com/apache/hadoop/pull/5517#issuecomment-1496234966
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 49s | | trunk passed |
| +1 :green_heart: | compile | 5m 59s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 5m 45s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 32s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 12s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 42s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 2s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | | the patch passed |
| +1 :green_heart: | compile | 5m 52s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 5m 52s | | the patch passed |
| +1 :green_heart: | compile | 5m 43s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 5m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 28s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 230m 49s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5517/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 34s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5517/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch failed. |
| +0 :ok: | asflicense | 0m 34s | | ASF License check generated no
output? |
| | | 380m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRead |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks |
| | hadoop.hdfs.TestMaintenanceWithStriped |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.namenode.TestReencryption |
| |
[jira] [Commented] (HDFS-16972) Delete a snapshot may deleteCurrentFile
[ https://issues.apache.org/jira/browse/HDFS-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708437#comment-17708437 ] ASF GitHub Bot commented on HDFS-16972: --- szetszwo opened a new pull request, #5532: URL: https://github.com/apache/hadoop/pull/5532 ### Description of PR We found one case the when deleting a snapshot (with ordered snapshot deletion disabled), it can incorrectly delete some files in the current state. ### How was this patch tested? Tested manually with a corrupted fsimage. Will see if we can add some new tests. ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [NA] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [NA] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Delete a snapshot may deleteCurrentFile > --- > > Key: HDFS-16972 > URL: https://issues.apache.org/jira/browse/HDFS-16972 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > > We found one case the when deleting a snapshot (with ordered snapshot > deletion disabled), it can incorrectly delete some files in the current state. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16972) Delete a snapshot may deleteCurrentFile
[ https://issues.apache.org/jira/browse/HDFS-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16972: -- Labels: pull-request-available (was: ) > Delete a snapshot may deleteCurrentFile > --- > > Key: HDFS-16972 > URL: https://issues.apache.org/jira/browse/HDFS-16972 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > > We found one case the when deleting a snapshot (with ordered snapshot > deletion disabled), it can incorrectly delete some files in the current state. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16972) Delete a snapshot may deleteCurrentFile
Tsz-wo Sze created HDFS-16972: - Summary: Delete a snapshot may deleteCurrentFile Key: HDFS-16972 URL: https://issues.apache.org/jira/browse/HDFS-16972 Project: Hadoop HDFS Issue Type: Bug Components: snapshots Reporter: Tsz-wo Sze Assignee: Tsz-wo Sze We found one case the when deleting a snapshot (with ordered snapshot deletion disabled), it can incorrectly delete some files in the current state. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16967) RBF: File based state stores should allow concurrent access to the records
[ https://issues.apache.org/jira/browse/HDFS-16967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16967. - Fix Version/s: 3.4.0 Resolution: Fixed > RBF: File based state stores should allow concurrent access to the records > -- > > Key: HDFS-16967 > URL: https://issues.apache.org/jira/browse/HDFS-16967 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > File based state store implementations (StateStoreFileImpl and > StateStoreFileSystemImpl) should allow updating as well as reading of the > state store records concurrently rather than serially. Concurrent access to > the record files on the hdfs based store seems to be improving the state > store cache loading performance by more than 10x. > For instance, in order to maintain data integrity, when any mount table > record(s) is updated, the cache is reloaded. This reload operation seems to > be able to gain significant performance improvement by the concurrent access > of the mount table records. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16967) RBF: File based state stores should allow concurrent access to the records
[ https://issues.apache.org/jira/browse/HDFS-16967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708428#comment-17708428 ] ASF GitHub Bot commented on HDFS-16967: --- tasanuma commented on PR #5523: URL: https://github.com/apache/hadoop/pull/5523#issuecomment-1495996505 Thanks for your contribution, @virajjasani. Thanks for reviewing it, @goiri and @simbadzina. > RBF: File based state stores should allow concurrent access to the records > -- > > Key: HDFS-16967 > URL: https://issues.apache.org/jira/browse/HDFS-16967 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > File based state store implementations (StateStoreFileImpl and > StateStoreFileSystemImpl) should allow updating as well as reading of the > state store records concurrently rather than serially. Concurrent access to > the record files on the hdfs based store seems to be improving the state > store cache loading performance by more than 10x. > For instance, in order to maintain data integrity, when any mount table > record(s) is updated, the cache is reloaded. This reload operation seems to > be able to gain significant performance improvement by the concurrent access > of the mount table records. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16967) RBF: File based state stores should allow concurrent access to the records
[ https://issues.apache.org/jira/browse/HDFS-16967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708427#comment-17708427 ] ASF GitHub Bot commented on HDFS-16967: --- tasanuma merged PR #5523: URL: https://github.com/apache/hadoop/pull/5523 > RBF: File based state stores should allow concurrent access to the records > -- > > Key: HDFS-16967 > URL: https://issues.apache.org/jira/browse/HDFS-16967 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > File based state store implementations (StateStoreFileImpl and > StateStoreFileSystemImpl) should allow updating as well as reading of the > state store records concurrently rather than serially. Concurrent access to > the record files on the hdfs based store seems to be improving the state > store cache loading performance by more than 10x. > For instance, in order to maintain data integrity, when any mount table > record(s) is updated, the cache is reloaded. This reload operation seems to > be able to gain significant performance improvement by the concurrent access > of the mount table records. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16952) Support getLinkTarget API in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-16952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708321#comment-17708321
]
ASF GitHub Bot commented on HDFS-16952:
---
hadoop-yetus commented on PR #5517:
URL: https://github.com/apache/hadoop/pull/5517#issuecomment-1495615197
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 48s | | trunk passed |
| +1 :green_heart: | compile | 6m 2s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 5m 48s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 6s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 30s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 49s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 41s | | the patch passed |
| +1 :green_heart: | compile | 5m 51s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 5m 51s | | the patch passed |
| +1 :green_heart: | compile | 5m 33s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 5m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 7s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5517/11/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 9 new + 146 unchanged - 0 fixed =
155 total (was 146) |
| +1 :green_heart: | mvnsite | 2m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 53s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 26s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 204m 5s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 20m 39s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 374m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5517/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5517 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux fb00d49d93fe 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
