[jira] [Commented] (HDFS-16876) Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools.
[
https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655656#comment-17655656
]
ASF GitHub Bot commented on HDFS-16876:
---
hadoop-yetus commented on PR #5282:
URL: https://github.com/apache/hadoop/pull/5282#issuecomment-1374389996
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 3
unchanged - 0 fixed = 4 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 37m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 142m 53s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5282 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4956f35a50cc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bcb5b8978d3d4001cc8b32aaf4f9664b2df57a41 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
[ https://issues.apache.org/jira/browse/HDFS-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655648#comment-17655648 ] ASF GitHub Bot commented on HDFS-16885: --- haiyang1987 opened a new pull request, #5283: URL: https://github.com/apache/hadoop/pull/5283 ### Description of PR [HDFS-16885](https://issues.apache.org/jira/browse/HDFS-16885) A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" was introduced in [HDFS-16881](https://issues.apache.org/jira/browse/HDFS-16881). However, this parameter was not added to hdfs-default.xml, cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. > Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed > --- > > Key: HDFS-16885 > URL: https://issues.apache.org/jira/browse/HDFS-16885 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" > was introduced in HDFS-16881. > However, this parameter was not added to hdfs-default.xml, cause run > TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
[ https://issues.apache.org/jira/browse/HDFS-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16885: -- Labels: pull-request-available (was: ) > Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed > --- > > Key: HDFS-16885 > URL: https://issues.apache.org/jira/browse/HDFS-16885 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" > was introduced in HDFS-16881. > However, this parameter was not added to hdfs-default.xml, cause run > TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
[ https://issues.apache.org/jira/browse/HDFS-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16885: -- Description: A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" was introduced in HDFS-16881. However, this parameter was not added to hdfs-default.xml, cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. was: A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" was introduced in HDFS-16881. However, this parameter was not added to hdfs-default.xml, cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. > Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed > --- > > Key: HDFS-16885 > URL: https://issues.apache.org/jira/browse/HDFS-16885 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" > was introduced in HDFS-16881. > However, this parameter was not added to hdfs-default.xml, cause run > TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
[ https://issues.apache.org/jira/browse/HDFS-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16885: -- Description: A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" was introduced in HDFS-16881. However, this parameter was not added to hdfs-default.xml, cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. > Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed > --- > > Key: HDFS-16885 > URL: https://issues.apache.org/jira/browse/HDFS-16885 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > A new parameter "dfs.namenode.access-control-enforcer-reporting-threshold-ms" > was introduced in HDFS-16881. > However, this parameter was not added to hdfs-default.xml, > cause run TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655639#comment-17655639 ] ASF GitHub Bot commented on HDFS-16884: --- haiyang1987 commented on PR #5280: URL: https://github.com/apache/hadoop/pull/5280#issuecomment-1374375708 Hi @ZanderXu @tomscut @tasanuma please help me review this pr when you are available, Thanks. > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > Since the default is async delete replica on the datanode, the replica may > not be deleted during the execution of UT#testConcurrentWriteAndDeleteBlock, > resulting in a mismatch between the number of replicas in each dataset > obtained at the end and the expectation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655638#comment-17655638 ] ASF GitHub Bot commented on HDFS-16884: --- haiyang1987 commented on PR #5280: URL: https://github.com/apache/hadoop/pull/5280#issuecomment-1374375533 The failed unit test seems unrelated to the change. > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > Since the default is async delete replica on the datanode, the replica may > not be deleted during the execution of UT#testConcurrentWriteAndDeleteBlock, > resulting in a mismatch between the number of replicas in each dataset > obtained at the end and the expectation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
[ https://issues.apache.org/jira/browse/HDFS-16885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu reassigned HDFS-16885: - Assignee: Haiyang Hu > Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed > --- > > Key: HDFS-16885 > URL: https://issues.apache.org/jira/browse/HDFS-16885 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16885) Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed
Haiyang Hu created HDFS-16885: - Summary: Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed Key: HDFS-16885 URL: https://issues.apache.org/jira/browse/HDFS-16885 Project: Hadoop HDFS Issue Type: Bug Reporter: Haiyang Hu -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16876) Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools.
[
https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655635#comment-17655635
]
ASF GitHub Bot commented on HDFS-16876:
---
hadoop-yetus commented on PR #5282:
URL: https://github.com/apache/hadoop/pull/5282#issuecomment-1374364662
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 6 new + 3
unchanged - 0 fixed = 9 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 36m 31s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 145m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5282 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 13b85b9e234d 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b0ea19c099af6859f4b69934ef9424f0045c61a2 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5282/1/testReport/ |
| Max. process+thread count | 2433 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Updated] (HDFS-16876) Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools.
[ https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16876: -- Labels: pull-request-available (was: ) > Garbage collect map entries in shared RouterStateIdContext using information > from namenodeResolver instead of the map of active connectionPools. > > > Key: HDFS-16876 > URL: https://issues.apache.org/jira/browse/HDFS-16876 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Critical > Labels: pull-request-available > > When cleaning up stale connection pools, the ConnectionManager also removes > the associated namespace to stateId mapping from the shared > RouterStateIdContext. A mapping is only removed when there is no other > connectionPool referencing it. When a new connection is created, the stateId > mapping gets recreated. > Since the RouterStateIdContext is shared across connections, it is cleaner to > drive its cleanup using the list of namespaces the router references instead > of closely tracking the active connections. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16876) Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools.
[ https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655614#comment-17655614 ] ASF GitHub Bot commented on HDFS-16876: --- simbadzina opened a new pull request, #5282: URL: https://github.com/apache/hadoop/pull/5282 HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. ### Description of PR Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data, versus tracking the set of active connection pools. ### How was this patch tested? New test cases in TestObserverWithRouter. **testSharedStateInRouterStateIdContext** validates that entries in the RouterStateIdContext are long lived and not recreated when connectionPools are deleted. This fails with the previous approach, and passes with the new one. **testRouterStateIdContextCleanup** validates that the new approach eventually deletes entries in the map. ### For code changes: - [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > Garbage collect map entries in shared RouterStateIdContext using information > from namenodeResolver instead of the map of active connectionPools. > > > Key: HDFS-16876 > URL: https://issues.apache.org/jira/browse/HDFS-16876 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Critical > > When cleaning up stale connection pools, the ConnectionManager also removes > the associated namespace to stateId mapping from the shared > RouterStateIdContext. A mapping is only removed when there is no other > connectionPool referencing it. When a new connection is created, the stateId > mapping gets recreated. > Since the RouterStateIdContext is shared across connections, it is cleaner to > drive its cleanup using the list of namespaces the router references instead > of closely tracking the active connections. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16876) Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools.
[ https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-16876: -- Summary: Garbage collect map entries in shared RouterStateIdContext using information from namenodeResolver instead of the map of active connectionPools. (was: ConnectionManager should not remove namespaceStateIds from the shared RouterStateIdContext during pool cleanup.) > Garbage collect map entries in shared RouterStateIdContext using information > from namenodeResolver instead of the map of active connectionPools. > > > Key: HDFS-16876 > URL: https://issues.apache.org/jira/browse/HDFS-16876 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Critical > > When cleaning up stale connection pools, the ConnectionManager also removes > the associated namespace to stateId mapping from the shared > RouterStateIdContext. A mapping is only removed when there is no other > connectionPool referencing it. When a new connection is created, the stateId > mapping gets recreated. > Since the RouterStateIdContext is shared across connections, it is cleaner to > drive its cleanup using the list of namespaces the router references instead > of closely tracking the active connections. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16876) ConnectionManager should not remove namespaceStateIds from the shared RouterStateIdContext during pool cleanup.
[ https://issues.apache.org/jira/browse/HDFS-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira updated HDFS-16876: -- Description: When cleaning up stale connection pools, the ConnectionManager also removes the associated namespace to stateId mapping from the shared RouterStateIdContext. A mapping is only removed when there is no other connectionPool referencing it. When a new connection is created, the stateId mapping gets recreated. Since the RouterStateIdContext is shared across connections, it is cleaner to drive its cleanup using the list of namespaces the router references instead of closely tracking the active connections. was: When cleaning up stale connection pools, the ConnectionManager also removes the associated namespaceStateId from the shared map in the RouterStateIdContext. This is incorrect because this namespaceStateId may still be referenced by other connection pools. The consequences of this premature removal include * No FederatedRouterState being sent back to clients. * PoolAlignmentContext objects keeping references to namespaceStateIds that aren't in the shared map. [https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java#L457] > ConnectionManager should not remove namespaceStateIds from the shared > RouterStateIdContext during pool cleanup. > --- > > Key: HDFS-16876 > URL: https://issues.apache.org/jira/browse/HDFS-16876 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Critical > > When cleaning up stale connection pools, the ConnectionManager also removes > the associated namespace to stateId mapping from the shared > RouterStateIdContext. A mapping is only removed when there is no other > connectionPool referencing it. When a new connection is created, the stateId > mapping gets recreated. > Since the RouterStateIdContext is shared across connections, it is cleaner to > drive its cleanup using the list of namespaces the router references instead > of closely tracking the active connections. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16877) Namenode doesn't use alignment context in TestObserverWithRouter
[
https://issues.apache.org/jira/browse/HDFS-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley resolved HDFS-16877.
--
Fix Version/s: 3.4.0
Assignee: Simbarashe Dzinamarira
Resolution: Fixed
I've committed this. Thanks, Simba!
> Namenode doesn't use alignment context in TestObserverWithRouter
>
>
> Key: HDFS-16877
> URL: https://issues.apache.org/jira/browse/HDFS-16877
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, rbf
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We need to set "{*}dfs.namenode.state.context.enabled{*}" to true for the
> namenode to send it's stateId in client responses.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16877) Namenode doesn't use alignment context in TestObserverWithRouter
[
https://issues.apache.org/jira/browse/HDFS-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655608#comment-17655608
]
ASF GitHub Bot commented on HDFS-16877:
---
omalley merged PR #5257:
URL: https://github.com/apache/hadoop/pull/5257
> Namenode doesn't use alignment context in TestObserverWithRouter
>
>
> Key: HDFS-16877
> URL: https://issues.apache.org/jira/browse/HDFS-16877
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, rbf
>Reporter: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
>
> We need to set "{*}dfs.namenode.state.context.enabled{*}" to true for the
> namenode to send it's stateId in client responses.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[
https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655508#comment-17655508
]
ASF GitHub Bot commented on HDFS-16884:
---
hadoop-yetus commented on PR #5280:
URL: https://github.com/apache/hadoop/pull/5280#issuecomment-1373875710
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 26s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 457m 30s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5280/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 574m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5280/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5280 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5b30496b5cd1 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9c2ed5fb9ee1ed812a4c76f85180f0a1aeb7276e |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5280/1/testReport/ |
| Max. process+thread count | 2014 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655503#comment-17655503
]
Simbarashe Dzinamarira commented on HDFS-13522:
---
No objection from me. Thanks for looking into them [~tasanuma]
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16877) Namenode doesn't use alignment context in TestObserverWithRouter
[
https://issues.apache.org/jira/browse/HDFS-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655501#comment-17655501
]
ASF GitHub Bot commented on HDFS-16877:
---
simbadzina commented on PR #5257:
URL: https://github.com/apache/hadoop/pull/5257#issuecomment-1373860804
@tomscut @goiri could either of you please merge this pull request for me.
> Namenode doesn't use alignment context in TestObserverWithRouter
>
>
> Key: HDFS-16877
> URL: https://issues.apache.org/jira/browse/HDFS-16877
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, rbf
>Reporter: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
>
> We need to set "{*}dfs.namenode.state.context.enabled{*}" to true for the
> namenode to send it's stateId in client responses.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655438#comment-17655438
]
Takanobu Asanuma commented on HDFS-13522:
-
Thanks for sharing it, [~simbadzina]. I looked over them, and it seems each
Jira is important, yet it keeps the backward compatibility. So we may be able
to backport them to branch-3.3. I'd like to create the parent Jira next week if
there is no objection.
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16863) Optimize frequency of regular block reports
[ https://issues.apache.org/jira/browse/HDFS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655400#comment-17655400 ] JiangHua Zhu commented on HDFS-16863: - [~yuyanlei], if FBR is reduced, will it have any new impact: 1. Some duplicates exist on Datanodes. NameNode should be notified but is not notified in time. 2. Complete the copy data saved by NameNode. > Optimize frequency of regular block reports > --- > > Key: HDFS-16863 > URL: https://issues.apache.org/jira/browse/HDFS-16863 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yanlei Yu >Priority: Major > Attachments: HDFS-16863.patch > > > like HDFS-15162 > Avoid sending block report at regular interval, if there is no failover, > DiskError or any exception encountered in connecting to the Namenode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
