[jira] [Commented] (HDFS-17316) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HDFS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17814042#comment-17814042
]
Han Liu commented on HDFS-17316:
Thanks for comments from [~ste...@apache.org]
{quote}If you look at the hadoop test classes
* we always need an explicit timeout
* on abfs/wasb/s3a tests we want to support parallel test execution, where you
can run multiple JNIT threads in parallel… each path and even temporary
directories must be uniquely allocated to the given thread, which we do by
passing a thread ID down.{quote}
Great suggestions. Im preparing the initial code, which has not take them into
account. I'll add to the doc and make a plan for update.
{quote} * In HADOOP-18508 I'm trying to support having different (local/remote)
hadoop source trees running test against the same S3 store. Goal: I can run a
hadoop public branch from one directory well debugging a different branch
locally. Something like that is needed here, given that it seems intended to
run against live HDFS clusters. Note that also brings authentication into the
mix, e.g. the option to use kinit to log in before running the tests. This is
not exclusive to HDFS either.{quote}
Exactly. All files/directories are created under the uri, so executions would
not affect each other if only the uri is uniquely generated.
For running tests with different hadoop branches against the same S3 cloud
service, my understanding is that this is achieved naturally as long as the uri
is uniquely generated each time.
{quote}It will be good -if not initially supported then at least designed as a
future option- to allow me to provide a list of stores to run the tests
against. This is because four S3A testing I now have to qualify with: amazon
s3, amazon s3 express, google gcs and at least one other implementation of the
API. All of which have slightly different capabilities -test process is going
to need to somehow be driven so that for the different implementation it knows
which features to test/results to expect. The current hadoop-aws/contract test
design is not up to this.
{quote}
Definitely. I personally run community supported S3A FS implementation. I am
glad that there would be many more targets so that we could comprehensively
evaluate the benchmark. However, only one FS instance can be supported per run.
This is because the benchmark should be simple and fast.
{quote}Microsoft are pushing hard at windows support. For the shell operations
it might be very good if rather than using bash/sh/zsh that python and pyunit
was the test runner, which could then invoke Windows commands as well as shall
scripts. Pyunit test report can be aggregated displayed in Jenkins, which is
another nice feature of them.
{quote}
Windows support is important while has not been taken into account. It is in
the future plan.
Some initial code would be finished soon, then I'll update a link.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HDFS-17316
> URL: https://issues.apache.org/jira/browse/HDFS-17316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS
[jira] [Updated] (HDFS-17371) Implement vectored io api
[ https://issues.apache.org/jira/browse/HDFS-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17371: -- Affects Version/s: 3.4.0 > Implement vectored io api > - > > Key: HDFS-17371 > URL: https://issues.apache.org/jira/browse/HDFS-17371 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: Jie Han >Priority: Major > > [HADOOP-18103|https://issues.apache.org/jira/browse/HADOOP-18103] introduced > a vectored io api which can significantly improve read performance, I hope > HDFS also implements it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17371) Implement vectored io api
[ https://issues.apache.org/jira/browse/HDFS-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17371: -- Component/s: dfsclient > Implement vectored io api > - > > Key: HDFS-17371 > URL: https://issues.apache.org/jira/browse/HDFS-17371 > Project: Hadoop HDFS > Issue Type: New Feature > Components: dfsclient >Affects Versions: 3.4.0 >Reporter: Jie Han >Priority: Major > > [HADOOP-18103|https://issues.apache.org/jira/browse/HADOOP-18103] introduced > a vectored io api which can significantly improve read performance, I hope > HDFS also implements it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17353: -- Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.5.0 >Reporter: Alexander Bogdanov >Assignee: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan reassigned HDFS-17353: - Assignee: Alexander Bogdanov > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.5.0 >Reporter: Alexander Bogdanov >Assignee: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813920#comment-17813920 ] Shilun Fan commented on HDFS-17353: --- [~ssanchozz] add to HDFS Contributor list, Welcome to Hadoop! > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.5.0 >Reporter: Alexander Bogdanov >Assignee: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17353: -- Affects Version/s: 3.5.0 > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.5.0 >Reporter: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17353: -- Target Version/s: 3.5.0 > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.5.0 >Reporter: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813919#comment-17813919 ] ASF GitHub Bot commented on HDFS-17353: --- slfan1989 commented on PR #6491: URL: https://github.com/apache/hadoop/pull/6491#issuecomment-1925290336 @ssanchozz Thanks for your contribution, merged into trunk. @goiri Thank you for reviewing the code! > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17353) Fix failing RBF module tests
[ https://issues.apache.org/jira/browse/HDFS-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813918#comment-17813918 ] ASF GitHub Bot commented on HDFS-17353: --- slfan1989 merged PR #6491: URL: https://github.com/apache/hadoop/pull/6491 > Fix failing RBF module tests > > > Key: HDFS-17353 > URL: https://issues.apache.org/jira/browse/HDFS-17353 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Alexander Bogdanov >Priority: Major > Labels: pull-request-available > > 3 tests are failing in RBF module: > * TestRouterRpcMultiDestination.testCallerContextWithMultiDestinations > * TestRouterRpcMultiDestination>TestRouterRpc.testMkdirsWithCallerContext > * > TestRouterRpcMultiDestination>TestRouterRpc.testRealUserPropagationInCallerContext > Need to fix them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813917#comment-17813917 ] Shilun Fan commented on HDFS-17369: --- [~haiyang Hu] When we open a new JIRA, Component/s should be filled in. targetVersion Version/s is 3.5.0. fixVersion needs to be filled in after the pr is merged. > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.5.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17369: -- Component/s: datanode > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813913#comment-17813913 ] ASF GitHub Bot commented on HDFS-17369: --- slfan1989 merged PR #6521: URL: https://github.com/apache/hadoop/pull/6521 > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813915#comment-17813915 ] ASF GitHub Bot commented on HDFS-17369: --- slfan1989 commented on PR #6521: URL: https://github.com/apache/hadoop/pull/6521#issuecomment-1925285005 @haiyang1987 thank you for your contribution! @ZanderXu Thanks for reviewing the code! > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.5.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17369: -- Target Version/s: 3.5.0 Affects Version/s: 3.5.0 > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.5.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17369) Add uuid into datanode info for NameNodeMXBean
[ https://issues.apache.org/jira/browse/HDFS-17369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved HDFS-17369. --- Fix Version/s: 3.5.0 Hadoop Flags: Reviewed Resolution: Fixed > Add uuid into datanode info for NameNodeMXBean > -- > > Key: HDFS-17369 > URL: https://issues.apache.org/jira/browse/HDFS-17369 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Add uuid into datanode info for NameNodeMXBean and convenient to locate > issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17342) Fix DataNode may invalidates normal block causing missing block
[ https://issues.apache.org/jira/browse/HDFS-17342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813906#comment-17813906 ] ASF GitHub Bot commented on HDFS-17342: --- haiyang1987 commented on PR #6464: URL: https://github.com/apache/hadoop/pull/6464#issuecomment-1925265679 Hi @zhangshuyan0 @ZanderXu @smarthanwang please help to push this modification forward If there are no any further comments. Thank you very much~ > Fix DataNode may invalidates normal block causing missing block > --- > > Key: HDFS-17342 > URL: https://issues.apache.org/jira/browse/HDFS-17342 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > When users read an append file, occasional exceptions may occur, such as > org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: xxx. > This can happen if one thread is reading the block while writer thread is > finalizing it simultaneously. > *Root cause:* > # The reader thread obtains a RBW replica from VolumeMap, such as: > blk_xxx_xxx[RBW] and the data file should be in /XXX/rbw/blk_xxx. > # Simultaneously, the writer thread will finalize this block, moving it from > the RBW directory to the FINALIZE directory. the data file is move from > /XXX/rbw/block_xxx to /XXX/finalize/block_xxx. > # The reader thread attempts to open this data input stream but encounters a > FileNotFoundException because the data file /XXX/rbw/blk_xxx or meta file > /XXX/rbw/blk_xxx_xxx doesn't exist at this moment. > # The reader thread will treats this block as corrupt, removes the replica > from the volume map, and the DataNode reports the deleted block to the > NameNode. > # The NameNode removes this replica for the block. > # If the current file replication is 1, this file will cause a missing block > issue until this DataNode executes the DirectoryScanner again. > As described above, when the reader thread encountered FileNotFoundException > is as expected, because the file is moved. > So we need to add a double check to the invalidateMissingBlock logic to > verify whether the data file or meta file exists to avoid similar cases. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17361) DiskBalancer: Query command support with multiple nodes
[ https://issues.apache.org/jira/browse/HDFS-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813905#comment-17813905 ] ASF GitHub Bot commented on HDFS-17361: --- haiyang1987 commented on PR #6508: URL: https://github.com/apache/hadoop/pull/6508#issuecomment-1925264228 Hi @zhangshuyan0 @ZanderXu @smarthanwang please help to push this modification forward If there are no any further comments. Thank you very much. > DiskBalancer: Query command support with multiple nodes > --- > > Key: HDFS-17361 > URL: https://issues.apache.org/jira/browse/HDFS-17361 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, diskbalancer >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > For: https://issues.apache.org/jira/browse/HDFS-10821 mentioned, Query > command will support with multiple nodes. > That means we can use command hdfs diskbalancer -query to print one or one > more datanodes status of the diskbalancer. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17361) DiskBalancer: Query command support with multiple nodes
[ https://issues.apache.org/jira/browse/HDFS-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813904#comment-17813904 ] ASF GitHub Bot commented on HDFS-17361: --- haiyang1987 commented on PR #6508: URL: https://github.com/apache/hadoop/pull/6508#issuecomment-1925260885 Hi @Hexiaoqiao @ZanderXu @zhangshuyan0 @tasanuma Could you mind help me review this PR when you have free time? Thanks~ > DiskBalancer: Query command support with multiple nodes > --- > > Key: HDFS-17361 > URL: https://issues.apache.org/jira/browse/HDFS-17361 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, diskbalancer >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > > For: https://issues.apache.org/jira/browse/HDFS-10821 mentioned, Query > command will support with multiple nodes. > That means we can use command hdfs diskbalancer -query to print one or one > more datanodes status of the diskbalancer. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17299) HDFS is not rack failure tolerant while creating a new file.
[
https://issues.apache.org/jira/browse/HDFS-17299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813900#comment-17813900
]
ASF GitHub Bot commented on HDFS-17299:
---
hadoop-yetus commented on PR #6513:
URL: https://github.com/apache/hadoop/pull/6513#issuecomment-1925254084
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 28s | | trunk passed |
| +1 :green_heart: | compile | 6m 6s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 55s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 1s | | the patch passed |
| +1 :green_heart: | compile | 5m 55s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 5m 55s | | the patch passed |
| +1 :green_heart: | compile | 5m 36s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 5m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 18s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6513/8/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 16 new + 243 unchanged - 2 fixed =
259 total (was 245) |
| +1 :green_heart: | mvnsite | 2m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 24s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 258m 1s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 445m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6513/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6513 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a59f38e0e351 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 04db2288b0964d44c7a918bc1324f95ee376f51b |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6513/8/testReport/ |
| Max. process+thread count
