[jira] [Commented] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[
https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676506#comment-17676506
]
ASF GitHub Bot commented on HDFS-16890:
---
hadoop-yetus commented on PR #5298:
URL: https://github.com/apache/hadoop/pull/5298#issuecomment-1381429142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 13s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 33m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 140m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5298 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 87ee841c1f74 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c2167ea3bf24437524dd792b0c83194a2d22bfc2 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
[jira] [Commented] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[ https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676496#comment-17676496 ] ASF GitHub Bot commented on HDFS-16890: --- hadoop-yetus commented on PR #5298: URL: https://github.com/apache/hadoop/pull/5298#issuecomment-1381388421 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 1m 28s | | Docker failed to build run-specific yetus/hadoop:tp-12069}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5298 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/4/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Add period state refresh to keep router state near active namenode's > > > Key: HDFS-16890 > URL: https://issues.apache.org/jira/browse/HDFS-16890 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > When using the ObserverReadProxyProvider, clients can set > *dfs.client.failover.observer.auto-msync-period...* to periodically get the > Active namenode's state. When using routers without the > ObserverReadProxyProvider, this periodic update is lost. > In a busy cluster, the Router constantly gets updated with the active > namenode's state when > # There is a write operation. > # There is an operation (read/write) from a new clients. > However, in the scenario when there are no new clients and no write > operations, the state kept in the router can lag behind the active's. The > router does update its state with responses from the Observer, but the > observer may be lagging behind too. > We should have a periodic refresh in the router to serve a similar role as > *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[
https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676485#comment-17676485
]
ASF GitHub Bot commented on HDFS-16890:
---
hadoop-yetus commented on PR #5298:
URL: https://github.com/apache/hadoop/pull/5298#issuecomment-1381374047
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 22m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 127m 36s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5298 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f91b575fa013 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bb0a34c030b7cf86c1fdcb00af957bc7ca14efd5 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
[jira] [Commented] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[ https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676472#comment-17676472 ] ASF GitHub Bot commented on HDFS-16890: --- hadoop-yetus commented on PR #5298: URL: https://github.com/apache/hadoop/pull/5298#issuecomment-1381358882 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | docker | 107m 55s | | Docker failed to build run-specific yetus/hadoop:tp-8622}. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5298 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5298/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Add period state refresh to keep router state near active namenode's > > > Key: HDFS-16890 > URL: https://issues.apache.org/jira/browse/HDFS-16890 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > When using the ObserverReadProxyProvider, clients can set > *dfs.client.failover.observer.auto-msync-period...* to periodically get the > Active namenode's state. When using routers without the > ObserverReadProxyProvider, this periodic update is lost. > In a busy cluster, the Router constantly gets updated with the active > namenode's state when > # There is a write operation. > # There is an operation (read/write) from a new clients. > However, in the scenario when there are no new clients and no write > operations, the state kept in the router can lag behind the active's. The > router does update its state with responses from the Observer, but the > observer may be lagging behind too. > We should have a periodic refresh in the router to serve a similar role as > *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[ https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16890: -- Labels: pull-request-available (was: ) > Add period state refresh to keep router state near active namenode's > > > Key: HDFS-16890 > URL: https://issues.apache.org/jira/browse/HDFS-16890 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > When using the ObserverReadProxyProvider, clients can set > *dfs.client.failover.observer.auto-msync-period...* to periodically get the > Active namenode's state. When using routers without the > ObserverReadProxyProvider, this periodic update is lost. > In a busy cluster, the Router constantly gets updated with the active > namenode's state when > # There is a write operation. > # There is an operation (read/write) from a new clients. > However, in the scenario when there are no new clients and no write > operations, the state kept in the router can lag behind the active's. The > router does update its state with responses from the Observer, but the > observer may be lagging behind too. > We should have a periodic refresh in the router to serve a similar role as > *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[ https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676432#comment-17676432 ] ASF GitHub Bot commented on HDFS-16890: --- simbadzina opened a new pull request, #5298: URL: https://github.com/apache/hadoop/pull/5298 HDFS-16890: Ensures router periodically refreshes its record of a namespace's state. ### Description of PR When using the ObserverReadProxyProvider, clients can set dfs.client.failover.observer.auto-msync-period... to periodically get the Active namenode's state. When using routers without the ObserverReadProxyProvider, this periodic update is lost. In a busy cluster, the Router constantly gets updated with the active namenode's state when There is a write operation. There is an operation (read/write) from a new clients. However, in the scenario when there are no new clients and no write operations, the state kept in the router can lag behind the active's. The router does update its state with responses from the Observer, but the observer may be lagging behind too. We should have a periodic refresh in the router to serve a similar role as dfs.client.failover.observer.auto-msync-period ### How was this patch tested? New test case. TestObserverWithRouter#testPeriodicStateRefreshUsingActiveNamenode() ### For code changes: - [ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? > Add period state refresh to keep router state near active namenode's > > > Key: HDFS-16890 > URL: https://issues.apache.org/jira/browse/HDFS-16890 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > > When using the ObserverReadProxyProvider, clients can set > *dfs.client.failover.observer.auto-msync-period...* to periodically get the > Active namenode's state. When using routers without the > ObserverReadProxyProvider, this periodic update is lost. > In a busy cluster, the Router constantly gets updated with the active > namenode's state when > # There is a write operation. > # There is an operation (read/write) from a new clients. > However, in the scenario when there are no new clients and no write > operations, the state kept in the router can lag behind the active's. The > router does update its state with responses from the Observer, but the > observer may be lagging behind too. > We should have a periodic refresh in the router to serve a similar role as > *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16890) Add period state refresh to keep router state near active namenode's
Simbarashe Dzinamarira created HDFS-16890: - Summary: Add period state refresh to keep router state near active namenode's Key: HDFS-16890 URL: https://issues.apache.org/jira/browse/HDFS-16890 Project: Hadoop HDFS Issue Type: Task Reporter: Simbarashe Dzinamarira When using the ObserverReadProxyProvider, clients can set *dfs.client.failover.observer.auto-msync-period...* to periodically get the Active namenode's state. When using routers without the ObserverReadProxyProvider, this periodic update is lost. In a busy cluster, the Router constantly gets updated with the active namenode's state when # There is a write operation. # There is an operation (read/write) from a new clients. However, in the scenario when there are no new clients and no write operations, the state kept in the router can lag behind the active's. The router does update its state with responses from the Observer, but the observer may be lagging behind too. We should have a periodic refresh in the router to serve a similar role as *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16890) Add period state refresh to keep router state near active namenode's
[ https://issues.apache.org/jira/browse/HDFS-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simbarashe Dzinamarira reassigned HDFS-16890: - Assignee: Simbarashe Dzinamarira > Add period state refresh to keep router state near active namenode's > > > Key: HDFS-16890 > URL: https://issues.apache.org/jira/browse/HDFS-16890 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > > When using the ObserverReadProxyProvider, clients can set > *dfs.client.failover.observer.auto-msync-period...* to periodically get the > Active namenode's state. When using routers without the > ObserverReadProxyProvider, this periodic update is lost. > In a busy cluster, the Router constantly gets updated with the active > namenode's state when > # There is a write operation. > # There is an operation (read/write) from a new clients. > However, in the scenario when there are no new clients and no write > operations, the state kept in the router can lag behind the active's. The > router does update its state with responses from the Observer, but the > observer may be lagging behind too. > We should have a periodic refresh in the router to serve a similar role as > *dfs.client.failover.observer.auto-msync-period* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676360#comment-17676360 ] ASF GitHub Bot commented on HDFS-16887: --- virajjasani commented on PR #5292: URL: https://github.com/apache/hadoop/pull/5292#issuecomment-1381067666 > The CI run had an unrelated test failure. Agree. Thank you for reviewing and merging the PR @cnauroth ! > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.9 > > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-16887. -- Fix Version/s: 3.4.0 3.2.5 3.3.9 Resolution: Fixed > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.5, 3.3.9 > > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676357#comment-17676357 ] ASF GitHub Bot commented on HDFS-16887: --- cnauroth commented on PR #5292: URL: https://github.com/apache/hadoop/pull/5292#issuecomment-1381066010 The CI run had an unrelated test failure. I have committed this to trunk, branch-3.3 and branch-3.2. @virajjasani , thank you for contributing the patch. > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-16887: - Component/s: namenode > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16887) Log start and end of phase/step in startup progress
[ https://issues.apache.org/jira/browse/HDFS-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676356#comment-17676356 ] ASF GitHub Bot commented on HDFS-16887: --- cnauroth merged PR #5292: URL: https://github.com/apache/hadoop/pull/5292 > Log start and end of phase/step in startup progress > --- > > Key: HDFS-16887 > URL: https://issues.apache.org/jira/browse/HDFS-16887 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > > As part of Namenode startup progress, we have multiple phases and steps > within phase that are instantiated. While the startup progress view can be > instantiated with the current view of phase/step, having at least DEBUG logs > for startup progress would be helpful to identify when a particular step for > LOADING_FSIMAGE/SAVING_CHECKPOINT/LOADING_EDITS was started and ended. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676240#comment-17676240
]
Takanobu Asanuma commented on HDFS-13522:
-
Created the umbrella Jira by HDFS-16889.
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16889) Backport JIRAs related to RBF SBN to branch-3.3
Takanobu Asanuma created HDFS-16889: --- Summary: Backport JIRAs related to RBF SBN to branch-3.3 Key: HDFS-16889 URL: https://issues.apache.org/jira/browse/HDFS-16889 Project: Hadoop HDFS Issue Type: Task Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma This is an umbrella JIRA to backport RBF SBN to branch-3.3. There are some conflicts when trying to backport HDFS-13522 and HDFS-16767, the main implementations of RBF SBN. Currently, to solve the conflicts, we need to backport the following JIRAs sequentially. (Thanks [~simbadzina] for the information.) # HDFS-14090 # HDFS-15417 # HDFS-16296 # HDFS-16302 # HDFS-15757 # HDFS-13274 # Then HDFS-13522 # HDFS-16065 # HDFS-16313 # HDFS-16273 # Then HDFS-16767 + other bug fixes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16888) BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile
[
https://issues.apache.org/jira/browse/HDFS-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676017#comment-17676017
]
ASF GitHub Bot commented on HDFS-16888:
---
hadoop-yetus commented on PR #5296:
URL: https://github.com/apache/hadoop/pull/5296#issuecomment-1380358113
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 43s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5296/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 85 unchanged -
2 fixed = 87 total (was 87) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 297m 15s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5296/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 414m 42s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5296/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5296 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 311e6babb525 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 49ab6d19c9fe69a79bda487a2668a39c38677230 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
[jira] [Commented] (HDFS-16569) Consider attaching block location info from client when closing a completed file
[
https://issues.apache.org/jira/browse/HDFS-16569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17675889#comment-17675889
]
Doris Gu commented on HDFS-16569:
-
> ` {{{}dfs.namenode.file.close.num-committed-allowed{}}}` is not helpful when
> most files' size is less than block size.
@[~yuanbo] I think ` {{{}dfs.namenode.file.close.num-committed-allowed{}}}`
is more helpful when most files' size is less than block size. Can you explain
why?
> Consider attaching block location info from client when closing a completed
> file
>
>
> Key: HDFS-16569
> URL: https://issues.apache.org/jira/browse/HDFS-16569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Major
>
> when a file is finished, client will not close it until DNs send
> RECEIVED_BLOCK by ibr or client is timeout. we can always see such kind of
> log in namenode
> {code:java}
> is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file{code}
> Since client already has the last block locations, it's not necessary to rely
> on the ibr from DN when closing file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
