[jira] [Comment Edited] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230394#comment-17230394
]
Fengnan Li edited comment on HDFS-14090 at 11/12/20, 7:07 AM:
--
Uploaded [^HDFS-14090.024.patch] to add configs for it.
I feel like there should be more optimization about how this config can be
specified and be made less verbose (like specify certain default values so we
don't need to specify all nameservices), but I cannot come up with a clean way
of doing this now. Will revisit when I start tackle the dynamic allocations.
Thanks!
was (Author: fengnanli):
Uploaded [^HDFS-14090.024.patch] to add configs for it.
I feel like there should be more optimization about how this config can be
specified but make it less verbose (like specify certain default values so we
don't need to specify all nameservices), but I cannot come up with a clean way
of doing this now. Will revisit when I start tackle the dynamic allocations.
Thanks!
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch,
> HDFS-14090.024.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230394#comment-17230394
]
Fengnan Li commented on HDFS-14090:
---
Uploaded [^HDFS-14090.024.patch] to add configs for it.
I feel like there should be more optimization about how this config can be
specified but make it less verbose (like specify certain default values so we
don't need to specify all nameservices), but I cannot come up with a clean way
of doing this now. Will revisit when I start tackle the dynamic allocations.
Thanks!
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch,
> HDFS-14090.024.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengnan Li updated HDFS-14090:
--
Attachment: HDFS-14090.024.patch
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch,
> HDFS-14090.024.patch, RBF_ Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-15538: Description: The description of dfs.namenode.replication.max-streams in hdfs-default.xml is misleading. The maxReplicationStreams is not limiting the replication streams with the highest priority. [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] was: The description of dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml is misleading. The maxReplicationStreams is not limiting the replication streams with the highest priority. [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch, HDFS-15538.002.patch > > > The description of dfs.namenode.replication.max-streams in hdfs-default.xml > is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-15538: Summary: Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml (was: Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml) > Fix the documentation for dfs.namenode.replication.max-streams in > hdfs-default.xml > -- > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch, HDFS-15538.002.patch > > > The description of dfs.namenode.replication.max-streams in hdfs-default.xml > is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230375#comment-17230375 ] Takanobu Asanuma commented on HDFS-15538: - [~risyomei] Thanks for updating the patch, and thanks for your message! +1 on [^HDFS-15538.002.patch], pending jenkins. > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch, HDFS-15538.002.patch > > > The description of dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-15538: -- Status: Open (was: Patch Available) > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch, HDFS-15538.002.patch > > > The description of dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xieming Li updated HDFS-15538: -- Attachment: HDFS-15538.002.patch Status: Patch Available (was: Open) > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch, HDFS-15538.002.patch > > > The description of dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230359#comment-17230359 ] Xieming Li edited comment on HDFS-15538 at 11/12/20, 5:27 AM: -- [~tasanuma] I apologize for not updating this ticket for a long time as I didn't noticed it. and Congratulations on your promotion to PMC! I have updated the patch base on your comment. was (Author: risyomei): [~tasanuma] I apologize for not updating this ticket for a long time as I didn't noticed it. and Congratulations on your promotion to PMC! I have updated the ticket base on your comment. > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch > > > The description of dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15538) Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230359#comment-17230359 ] Xieming Li commented on HDFS-15538: --- [~tasanuma] I apologize for not updating this ticket for a long time as I didn't noticed it. and Congratulations on your promotion to PMC! I have updated the ticket base on your comment. > Fix the documentation for dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml > - > > Key: HDFS-15538 > URL: https://issues.apache.org/jira/browse/HDFS-15538 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Labels: documentation > Attachments: HDFS-15538.001.patch > > > The description of dfs.namenode.replication.max-streams-hard-limit in > hdfs-default.xml is misleading. > The maxReplicationStreams is not limiting the replication streams with the > highest priority. > [https://github.com/apache/hadoop/blob/branch-3.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L2463-L2471] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15681) TestDynamometerInfra fails
Akira Ajisaka created HDFS-15681:
Summary: TestDynamometerInfra fails
Key: HDFS-15681
URL: https://issues.apache.org/jira/browse/HDFS-15681
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Akira Ajisaka
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/321/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt
{noformat}
[INFO] Running org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.43 s
<<< FAILURE! - in org.apache.hadoop.tools.dynamometer.TestDynamometerInfra
[ERROR] org.apache.hadoop.tools.dynamometer.TestDynamometerInfra Time elapsed:
0.43 s <<< ERROR!
java.io.FileNotFoundException:
http://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at org.apache.commons.io.FileUtils.copyURLToFile(FileUtils.java:1506)
at
org.apache.hadoop.tools.dynamometer.DynoInfraUtils.fetchHadoopTarball(DynoInfraUtils.java:151)
at
org.apache.hadoop.tools.dynamometer.TestDynamometerInfra.setupClass(TestDynamometerInfra.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230347#comment-17230347
]
Yiqun Lin commented on HDFS-14090:
--
Sounds good to me, let's address #2 comment, [~fengnanli].
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230344#comment-17230344
]
Fengnan Li commented on HDFS-14090:
---
Thanks for the review [~linyiqun] and here is the late response.
1. the isolated concurrent is for the case where renew lease doesn't stop due
to connection leak. We have seen this production and YARN added some fix to it.
Interestingly, the bug was fixed by you in
[HDFS-10549|https://issues.apache.org/jira/browse/HDFS-10549] . If we remove
concurrent, these calls would consume the permit for other calls quickly and
soon the whole cluster will be swamped with renew lease.
I will address 2 in a following patch and leave 3 as a follow-up JIRA when more
production data is collected.
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15680) Disable Broken Azure Junits
[
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230333#comment-17230333
]
Ahmed Hussein edited comment on HDFS-15680 at 11/12/20, 2:59 AM:
-
{quote}FWIW I hadn't noticed this until last week. This is the kind of thing to
raise on common dev, rather than just say "let's turn off these tests"{quote}
[~ste...@apache.org] Thanks for fixing those tests.
There have been several emails sent, over the last month, regarding the failing
tests and Yetus failures.
[~ayushtkn], [~elgoiri], [~weichiu], and [~aajisaka] responded to those emails.
was (Author: ahussein):
{quote}FWIW I hadn't noticed this until last week. This is the kind of thing to
raise on common dev, rather than just say "let's turn off these tests"{quote}
[~ste...@apache.org] Thanks for fixing those tests.
There have been several emails sent, over the last month, regarding the failing
tests and Yetus failures status.
[~ayushtkn], [~elgoiri], [~weichiu], and [~aajisaka] responded to those emails.
> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months.
> They contributed to more than 41 failing tests which makes reviewing Yetus
> reports every a pain in the neck. Another point is to save the resources and
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific
> case, I do not assume that it makes any sense to have 41 failing tests from
> one module for several months. Whenever someone finds that those tests are
> useful, then they can re-enable the tests on Yetus *_After_* the test is
> fixed.
> Following a PR, I have to review that my patch does not cause any failures
> (include changing error messages in existing tests). A thorough review takes
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports,
> because it would take too much time to review; and by default, the errors are
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
> hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
>hadoop.fs.azure.TestNativeAzureFileSystemMocked
>hadoop.fs.azure.TestBlobMetadata
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
>hadoop.fs.azure.TestWasbFsck
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
>
[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits
[
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230333#comment-17230333
]
Ahmed Hussein commented on HDFS-15680:
--
{quote}FWIW I hadn't noticed this until last week. This is the kind of thing to
raise on common dev, rather than just say "let's turn off these tests"{quote}
[~ste...@apache.org] Thanks for fixing those tests.
There have been several emails sent, over the last month, regarding the failing
tests and Yetus failures status.
[~ayushtkn], [~elgoiri], [~weichiu], and [~aajisaka] responded to those emails.
> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months.
> They contributed to more than 41 failing tests which makes reviewing Yetus
> reports every a pain in the neck. Another point is to save the resources and
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific
> case, I do not assume that it makes any sense to have 41 failing tests from
> one module for several months. Whenever someone finds that those tests are
> useful, then they can re-enable the tests on Yetus *_After_* the test is
> fixed.
> Following a PR, I have to review that my patch does not cause any failures
> (include changing error messages in existing tests). A thorough review takes
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports,
> because it would take too much time to review; and by default, the errors are
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
> hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
>hadoop.fs.azure.TestNativeAzureFileSystemMocked
>hadoop.fs.azure.TestBlobMetadata
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
>hadoop.fs.azure.TestWasbFsck
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
>
[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits
[
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230224#comment-17230224
]
Steve Loughran commented on HDFS-15680:
---
bq. here are 6 test classes have been failing on Yetus for several months.
FWIW I hadn't noticed this until last week. This is the kind of thing to raise
on common dev, rather than just say "let's turn off these tests"
> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months.
> They contributed to more than 41 failing tests which makes reviewing Yetus
> reports every a pain in the neck. Another point is to save the resources and
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific
> case, I do not assume that it makes any sense to have 41 failing tests from
> one module for several months. Whenever someone finds that those tests are
> useful, then they can re-enable the tests on Yetus *_After_* the test is
> fixed.
> Following a PR, I have to review that my patch does not cause any failures
> (include changing error messages in existing tests). A thorough review takes
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports,
> because it would take too much time to review; and by default, the errors are
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
> hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
>hadoop.fs.azure.TestNativeAzureFileSystemMocked
>hadoop.fs.azure.TestBlobMetadata
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
>hadoop.fs.azure.TestWasbFsck
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
>
[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits
[
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230222#comment-17230222
]
Steve Loughran commented on HDFS-15680:
---
-1.
These tests are working: they have found a regression caused by a JAR update
> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months.
> They contributed to more than 41 failing tests which makes reviewing Yetus
> reports every a pain in the neck. Another point is to save the resources and
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific
> case, I do not assume that it makes any sense to have 41 failing tests from
> one module for several months. Whenever someone finds that those tests are
> useful, then they can re-enable the tests on Yetus *_After_* the test is
> fixed.
> Following a PR, I have to review that my patch does not cause any failures
> (include changing error messages in existing tests). A thorough review takes
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports,
> because it would take too much time to review; and by default, the errors are
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
> hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
>hadoop.fs.azure.TestNativeAzureFileSystemMocked
>hadoop.fs.azure.TestBlobMetadata
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
>hadoop.fs.azure.TestWasbFsck
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked.testListStatus
>
[jira] [Work logged] (HDFS-15635) ViewFileSystemOverloadScheme support specifying mount table loader imp through conf
[
https://issues.apache.org/jira/browse/HDFS-15635?focusedWorklogId=510420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-510420
]
ASF GitHub Bot logged work on HDFS-15635:
-
Author: ASF GitHub Bot
Created on: 11/Nov/20 19:40
Start Date: 11/Nov/20 19:40
Worklog Time Spent: 10m
Work Description: umamaheswararao commented on pull request #2389:
URL: https://github.com/apache/hadoop/pull/2389#issuecomment-725622153
@zuston Could you please check the checkstyle warnings seems related?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 510420)
Time Spent: 1h (was: 50m)
> ViewFileSystemOverloadScheme support specifying mount table loader imp
> through conf
> ---
>
> Key: HDFS-15635
> URL: https://issues.apache.org/jira/browse/HDFS-15635
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfsOverloadScheme
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> According to HDFS-15289, the default mountable loader is
> {{[HCFSMountTableConfigLoader|https://github.com/apache/hadoop/blob/4734c77b4b64b7c6432da4cc32881aba85f94ea1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/HCFSMountTableConfigLoader.java#L35]}}.
> In some scenarios, users want to implement the mount table loader by
> themselves, so it is necessary to dynamically configure the loader.
>
> cc [~shv], [~abhishekd], [~hexiaoqiao]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15659) Set dfs.namenode.redundancy.considerLoad to false in MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-15659?focusedWorklogId=510404=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-510404
]
ASF GitHub Bot logged work on HDFS-15659:
-
Author: ASF GitHub Bot
Created on: 11/Nov/20 19:15
Start Date: 11/Nov/20 19:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2443:
URL: https://github.com/apache/hadoop/pull/2443#issuecomment-725609791
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 13 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 32s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 3m 15s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 13s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 39s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | findbugs | 3m 13s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 116m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 213m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2443 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 36d402169250 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e6d2dccbef6 |
| Default Java | Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2443/3/testReport/ |
| Max. process+thread count | 2653 (vs. ulimit of 5500) |
|
[jira] [Updated] (HDFS-15296) TestBPOfferService#testMissBlocksWhenReregister is flaky
[
https://issues.apache.org/jira/browse/HDFS-15296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated HDFS-15296:
-
Summary: TestBPOfferService#testMissBlocksWhenReregister is flaky (was:
TestBPOfferService::testMissBlocksWhenReregister is flaky)
> TestBPOfferService#testMissBlocksWhenReregister is flaky
>
>
> Key: HDFS-15296
> URL: https://issues.apache.org/jira/browse/HDFS-15296
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: 3.4.0
>Reporter: Mingliang Liu
>Priority: Major
>
> TestBPOfferService.testMissBlocksWhenReregister fails intermittently in
> {{trunk}} branch, not sure about other branches. Example failures are
> -
> https://builds.apache.org/job/hadoop-multibranch/job/PR-1964/4/testReport/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> -
> https://builds.apache.org/job/PreCommit-HDFS-Build/29175/testReport/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> Sample exception stack is:
> {quote}
> Stacktrace
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15680) Disable Broken Azure Junits
[
https://issues.apache.org/jira/browse/HDFS-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229900#comment-17229900
]
Xiaoqiao He commented on HDFS-15680:
Thanks [~ayushtkn] for involve me here. [~aajisaka] and [~weichiu] may be more
proper to confirm it. If we need to revert HADOOP-17223, I would like to
prepare another RC for 3.2.2 release. Thanks.
> Disable Broken Azure Junits
> ---
>
> Key: HDFS-15680
> URL: https://issues.apache.org/jira/browse/HDFS-15680
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are 6 test classes have been failing on Yetus for several months.
> They contributed to more than 41 failing tests which makes reviewing Yetus
> reports every a pain in the neck. Another point is to save the resources and
> avoiding utilization of ports, memory, and CPU.
> Over the last month, there was some effort to bring the Yetus back to a
> stable state. However, there is no progress in addressing Azure failures.
> Generally, I do not like to disable failing tests, but for this specific
> case, I do not assume that it makes any sense to have 41 failing tests from
> one module for several months. Whenever someone finds that those tests are
> useful, then they can re-enable the tests on Yetus *_After_* the test is
> fixed.
> Following a PR, I have to review that my patch does not cause any failures
> (include changing error messages in existing tests). A thorough review takes
> a considerable amount of time browsing the nightly builds and Github reports.
> So, please consider how much time is being spent to review those stack trace
> over the last months.
> Finally, this is one of the reasons developers tend to ignore the reports,
> because it would take too much time to review; and by default, the errors are
> considered irrelevant.
> CC: [~aajisaka], [~elgoiri], [~weichiu], [~ayushtkn]
> {code:bash}
> hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
>hadoop.fs.azure.TestNativeAzureFileSystemMocked
>hadoop.fs.azure.TestBlobMetadata
>hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
>hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
>hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
>hadoop.fs.azure.TestWasbFsck
>hadoop.fs.azure.TestOutOfBandAzureBlobOperations
> {code}
> {code:bash}
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatusRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryMoveToExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testListStatus
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameDirectoryAsExistingDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testRenameToDirWithSamePrefixAllowed
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testLSRootDir
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked.testDeleteRecursively
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck.testWasbFsck
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testChineseCharactersFolderRename
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderInFolderListing
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testUriEncoding
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testDeepFileCreation
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testListDirectory
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolderRenameInProgress
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRenameImplicitFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testRedoRenameFolder
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked.testStoreDeleteFolder
>
