[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[Takanobu Asanuma (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692547#comment-17692547
 ] 

Takanobu Asanuma commented on HDFS-16845:
-

It is only in trunk. I corrected the fixed versions.

> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17640317#comment-17640317
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

omalley merged PR #5142:
URL: https://github.com/apache/hadoop/pull/5142




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17638837#comment-17638837
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1328020273

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 42s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 54s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m 11s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m 21s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   6m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 28s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   3m 57s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m  1s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 40s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 39s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 13s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 54s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m  4s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   6m  4s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 51s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   5m 51s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   1m 12s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   3m 17s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 11s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 15s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 30s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m  9s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 33s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | -1 :x: |  unit  | 301m 28s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | -1 :x: |  unit  |  23m  4s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
 |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m  4s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 481m  4s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
   |   | hadoop.hdfs.server.balancer.TestBalancerService |
   |   | 
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
 |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/7/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux a9b06287bb2f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17638159#comment-17638159
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1326085520

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 39s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 57s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m  6s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m 15s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   5m 54s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 24s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   3m 47s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m  0s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 41s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 38s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m  1s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 32s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 54s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m  0s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   6m  0s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 44s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   5m 44s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 14s | 
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/6/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
 |  hadoop-hdfs-project: The patch generated 1 new + 33 unchanged - 0 fixed = 
34 total (was 33)  |
   | +1 :green_heart: |  mvnsite  |   3m  5s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 10s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m  5s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 31s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 51s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 41s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | -1 :x: |  unit  | 306m 17s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  unit  |  23m  5s |  |  hadoop-hdfs-rbf in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   1m 18s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 484m 37s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/6/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux 164b9b4c8623 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637474#comment-17637474
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1324273744

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 38s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 44s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m 10s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m 14s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   5m 54s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 34s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   3m 50s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   2m 58s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   4m  1s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   9m  6s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 27s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 31s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 54s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m  5s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   6m  5s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 41s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   5m 41s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 15s | 
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/5/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
 |  hadoop-hdfs-project: The patch generated 1 new + 33 unchanged - 0 fixed = 
34 total (was 33)  |
   | +1 :green_heart: |  mvnsite  |   3m  8s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 22s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 10s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 27s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m 50s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 32s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | -1 :x: |  unit  | 281m 46s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | -1 :x: |  unit  |  23m 17s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
 |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m  2s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 465m 14s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
   |   | 
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
 |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/5/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637369#comment-17637369
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1029625039


##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,13 @@ public static ClientProtocol 
createProxyWithAlignmentContext(
   boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
   AlignmentContext alignmentContext)
   throws IOException {
+if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,

Review Comment:
   Resolved.





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637156#comment-17637156
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1323455565

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 40s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  20m 10s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 32s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   7m 23s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   7m  6s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 35s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m  8s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m 11s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   8m 15s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 15s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 32s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   3m 11s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   7m 17s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   7m 17s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m 46s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   6m 46s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 17s | 
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/4/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
 |  hadoop-hdfs-project: The patch generated 2 new + 33 unchanged - 0 fixed = 
35 total (was 33)  |
   | +1 :green_heart: |  mvnsite  |   3m 17s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 23s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m  9s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 52s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m  9s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 41s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | -1 :x: |  unit  | 304m 38s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  unit  |  34m 39s |  |  hadoop-hdfs-rbf in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   1m 13s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 507m 54s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/4/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux 15a18a4df778 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637152#comment-17637152
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1323448865

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 57s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  18m 12s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 32s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m 48s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   7m 14s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 52s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   3m 50s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m 17s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 58s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   8m 23s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 29s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 29s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 58s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m 49s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   6m 49s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   6m 54s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   6m 54s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 19s | 
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/3/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
 |  hadoop-hdfs-project: The patch generated 2 new + 33 unchanged - 0 fixed = 
35 total (was 33)  |
   | +1 :green_heart: |  mvnsite  |   3m 38s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 16s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   8m  0s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m  7s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 30s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | -1 :x: |  unit  | 315m 34s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  unit  |  23m 33s |  |  hadoop-hdfs-rbf in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   1m 10s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 506m  5s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/3/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux 3503756845d9 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636944#comment-17636944
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

omalley commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028642292


##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,13 @@ public static ClientProtocol 
createProxyWithAlignmentContext(
   boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
   AlignmentContext alignmentContext)
   throws IOException {
+if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,

Review Comment:
   I'd simplify the check to
   ```
   if (alignmentContext == null &&
  conf.getBoolean()) {
 alignmentContext = new ClientGSIContext();
   }
   ```





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636943#comment-17636943
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

omalley commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028642292


##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,13 @@ public static ClientProtocol 
createProxyWithAlignmentContext(
   boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
   AlignmentContext alignmentContext)
   throws IOException {
+if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,

Review Comment:
   I'd simplify the check to
   if (alignmentContext == null &&
  conf.getBoolean()) {
 alignmentContext = new ClientGSIContext();
   }





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636896#comment-17636896
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1322676213

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 42s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 40s |  |  Maven dependency ordering for branch  |
   | -1 :x: |  mvninstall  |  23m 21s | 
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-mvninstall-root.txt)
 |  root in trunk failed.  |
   | -1 :x: |  compile  |   1m 45s | 
[/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
 |  hadoop-hdfs-project in trunk failed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  compile  |   0m 52s | 
[/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-compile-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
 |  hadoop-hdfs-project in trunk failed with JDK Private 
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.  |
   | +1 :green_heart: |  checkstyle  |   1m 52s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   4m 17s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m 27s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   4m  8s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | -1 :x: |  spotbugs  |   3m 24s | 
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
 |  hadoop-hdfs-client in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 47s | 
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in trunk failed.  |
   | -1 :x: |  spotbugs  |   0m 48s | 
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
 |  hadoop-hdfs-rbf in trunk failed.  |
   | +1 :green_heart: |  shadedclient  |  20m 37s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 28s |  |  Maven dependency ordering for patch  |
   | -1 :x: |  mvninstall  |   0m 32s | 
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 30s | 
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt)
 |  hadoop-hdfs-client in the patch failed.  |
   | -1 :x: |  mvninstall  |   0m 31s | 
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
 |  hadoop-hdfs-rbf in the patch failed.  |
   | -1 :x: |  compile  |   0m 32s | 
[/patch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/2/artifact/out/patch-compile-hadoop-hdfs-project-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
 |  hadoop-hdfs-project in the patch failed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.  |
   | -1 :x: |  javac  |   0m 32s | 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636872#comment-17636872
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028476373


##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,13 @@ public static ClientProtocol 
createProxyWithAlignmentContext(
   boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
   AlignmentContext alignmentContext)
   throws IOException {
+if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,

Review Comment:
   If someone passes in null, then the client will not echo back alignment 
state to the router, which it turn makes the router always forward these calls 
to the active namenode.





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636871#comment-17636871
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028474832


##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java:
##
@@ -439,4 +440,60 @@ public void testRouterMsync() throws Exception {
 assertEquals("Four calls should be sent to active", 4,
 rpcCountForActive);
   }
+
+  @Test
+  public void testSingleRead() throws Exception {
+List namenodes = routerContext
+.getRouter().getNamenodeResolver()
+.getNamenodesForNameserviceId(cluster.getNameservices().get(0), true);
+assertEquals("First namenode should be observer", 
namenodes.get(0).getState(),
+FederationNamenodeServiceState.OBSERVER);
+Path path = new Path("/");
+
+long rpcCountForActive;
+long rpcCountForObserver;
+
+// Send read request
+fileSystem.listFiles(path, false);
+fileSystem.close();
+
+rpcCountForActive = routerContext.getRouter().getRpcServer()
+.getRPCMetrics().getActiveProxyOps();
+// getListingCall sent to active.
+assertEquals("Only one call should be sent to active", 1, 
rpcCountForActive);
+
+rpcCountForObserver = routerContext.getRouter().getRpcServer()
+.getRPCMetrics().getObserverProxyOps();
+// getList call should be sent to observer
+assertEquals("No calls should be sent to observer", 0, 
rpcCountForObserver);
+  }
+
+  @Test
+  public void testSingleReadUsingObserverReadProxyProvider() throws Exception {
+fileSystem.close();
+fileSystem = routerContext.getFileSystemWithObserverReadProxyProvider();

Review Comment:
   I agree. I'm now moving the code into the individual tests.





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636813#comment-17636813
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

mkuchenbecker commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028371062


##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java:
##
@@ -122,7 +123,9 @@ public void startUpCluster(int numberOfObserver, 
Configuration confOverrides) th
 
 cluster.waitActiveNamespaces();
 routerContext  = cluster.getRandomRouter();
-fileSystem = routerContext.getFileSystemWithObserverReadsEnabled();
+Configuration confToEnableObserverRead = new Configuration();
+
confToEnableObserverRead.setBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,
 true);
+fileSystem = routerContext.getFileSystem(confToEnableObserverRead);

Review Comment:
   We are losing coverage on `getFileSystemWithObserverReadsEnabled` with this 
change; we should likely be testing both as they are both valid use-cases 
whether you want to msync or not.





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636802#comment-17636802
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

mkuchenbecker commented on code in PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#discussion_r1028372645


##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java:
##
@@ -439,4 +440,60 @@ public void testRouterMsync() throws Exception {
 assertEquals("Four calls should be sent to active", 4,
 rpcCountForActive);
   }
+
+  @Test
+  public void testSingleRead() throws Exception {
+List namenodes = routerContext
+.getRouter().getNamenodeResolver()
+.getNamenodesForNameserviceId(cluster.getNameservices().get(0), true);
+assertEquals("First namenode should be observer", 
namenodes.get(0).getState(),
+FederationNamenodeServiceState.OBSERVER);
+Path path = new Path("/");
+
+long rpcCountForActive;
+long rpcCountForObserver;
+
+// Send read request
+fileSystem.listFiles(path, false);
+fileSystem.close();
+
+rpcCountForActive = routerContext.getRouter().getRpcServer()
+.getRPCMetrics().getActiveProxyOps();
+// getListingCall sent to active.
+assertEquals("Only one call should be sent to active", 1, 
rpcCountForActive);
+
+rpcCountForObserver = routerContext.getRouter().getRpcServer()
+.getRPCMetrics().getObserverProxyOps();
+// getList call should be sent to observer
+assertEquals("No calls should be sent to observer", 0, 
rpcCountForObserver);
+  }
+
+  @Test
+  public void testSingleReadUsingObserverReadProxyProvider() throws Exception {
+fileSystem.close();
+fileSystem = routerContext.getFileSystemWithObserverReadProxyProvider();

Review Comment:
   This seems wrong to special-case in this way. Either manage it during setup 
or set it for every function, but I'd advise against mixing the two.



##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestObserverWithRouter.java:
##
@@ -122,7 +123,9 @@ public void startUpCluster(int numberOfObserver, 
Configuration confOverrides) th
 
 cluster.waitActiveNamespaces();
 routerContext  = cluster.getRandomRouter();
-fileSystem = routerContext.getFileSystemWithObserverReadsEnabled();
+Configuration confToEnableObserverRead = new Configuration();
+
confToEnableObserverRead.setBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,
 true);
+fileSystem = routerContext.getFileSystem(confToEnableObserverRead);

Review Comment:
   We are losing coverage on `getFileSystemWithObserverReadsEnabled` with this 
change; we should likely be testing both as they are both valid use-cases 
whether you want to msync or not.



##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,13 @@ public static ClientProtocol 
createProxyWithAlignmentContext(
   boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
   AlignmentContext alignmentContext)
   throws IOException {
+if (conf.getBoolean(HdfsClientConfigKeys.DFS_RBF_OBSERVER_READ_ENABLE,

Review Comment:
   What was the original behaviour where someone passed in `null` to this 
function? 





> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636801#comment-17636801
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

mkuchenbecker commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1322457075

   Mostly mechanical review, with some questions. 




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635652#comment-17635652
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1319526622

   > > @ZanderXu @tomscut could you please take a look whenever you get a 
chance.
   > 
   > @simbadzina I did a quick review. The change looks good to me. I am 
curious about that what scenarios will use this feature without `msyc()`. Can 
you share me some cases?
   
   Thanks @ZanderXu . Third party readers issuing multiple reads will use msync 
if they want consistency, both before and after this path. The msync this flag 
eliminates is the one at the initialization of the ObserverReadProxyProvider 
here
   
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java#L341-L348
   
   Taking an example of a client just doing a list status.
   
   If the client is using ObserverReadProxyProvider. They'll be two calls.
   1) msync (during initialization)
   2) getListing (actual call the client wants to do)
   When the router gets the msync, it will fan this out to all namespace, which 
is expensive.
   
   With this patch, they'll only be one call.
   1) getListing
   The router will send this call to the active.
   
   For future reads after this, there is no difference.




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635631#comment-17635631
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

ZanderXu commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1319492306

   > @ZanderXu @tomscut could you please take a look whenever you get a chance.
   
   @simbadzina I did a quick review. The change looks good to me. I am curious 
about that what scenarios will use this feature without `msyc()`. Can you share 
me some cases?




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635627#comment-17635627
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1319486380

   @ZanderXu @tomscut could you please take a look whenever you get a chance.




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Critical
>  Labels: pull-request-available
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespaces, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635275#comment-17635275
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

hadoop-yetus commented on PR #5142:
URL: https://github.com/apache/hadoop/pull/5142#issuecomment-1318402899

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 44s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +0 :ok: |  xmllint  |   0m  1s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  15m 56s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  27m 19s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m 31s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   6m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 31s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   3m 59s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   3m  1s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 41s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 54s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 50s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 33s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m 51s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 58s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   5m 58s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 57s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   5m 57s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   1m 15s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   3m 10s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   2m 22s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   3m 12s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   7m 54s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  22m 29s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 41s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | +1 :green_heart: |  unit  | 250m 31s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  unit  |  23m  5s |  |  hadoop-hdfs-rbf in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   1m 13s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 434m 38s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5142 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux 145b5c8046bf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 668d5d217d8386e936246caabeafa7f50c05b10f |
   | Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5142/1/testReport/ |
   | Max. process+thread count | 3386 (vs. ulimit of 

[jira] [Commented] (HDFS-16845) Add configuration flag to enable observer reads on routers without using ObserverReadProxyProvider

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635091#comment-17635091
 ] 

ASF GitHub Bot commented on HDFS-16845:
---

simbadzina opened a new pull request, #5142:
URL: https://github.com/apache/hadoop/pull/5142

   HDFS-16845: Adds configuration flag to allow clients to use router observer 
reads without using the ObserverReadProxyProvider.
   
   
   
   ### Description of PR
   Description
   In order for clients to have routers forward their reads to observers, the 
clients must use a proxy with an alignment context. This is currently achieved 
by using the ObserverReadProxyProvider.
   
   Using ObserverReadProxyProvider allows backward compatible for client 
configurations.
   However, the ObserverReadProxyProvider forces an msync on initialization 
which is not required with routers.
   
   Performing msync calls is more expensive with routers because the router 
fans out the cal to all namespace, so we'd like to avoid this.
   
   ### How was this patch tested?
   New test cases in TestObserverWithRouter.
   
   ### For code changes:
   
   - [ x] Does the title or this PR starts with the corresponding JIRA issue id 
(e.g. 'HADOOP-17799. Your PR title ...')?
   




> Add configuration flag to enable observer reads on routers without using 
> ObserverReadProxyProvider
> --
>
> Key: HDFS-16845
> URL: https://issues.apache.org/jira/browse/HDFS-16845
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
>
> In order for clients to have routers forward their reads to observers, the 
> clients must use a proxy with an alignment context. This is currently 
> achieved by using the ObserverReadProxyProvider.
> Using ObserverReadProxyProvider allows backward compatible for client 
> configurations.
> However, the ObserverReadProxyProvider forces an msync on initialization 
> which is not required with routers.
> Performing msync calls is more expensive with routers because the router fans 
> out the cal to all namespace, so we'd like to avoid this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org