date:20240220

[jira] [Resolved] (HDFS-17181) WebHDFS not considering whether a DN is good when called from outside the cluster

2024-02-20 Thread Wei-Chiu Chuang (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-17181.

Fix Version/s: 3.5.0
   Resolution: Fixed

> WebHDFS not considering whether a DN is good when called from outside the 
> cluster
> -
>
> Key: HDFS-17181
> URL: https://issues.apache.org/jira/browse/HDFS-17181
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode, webhdfs
>Affects Versions: 3.3.6
>Reporter: Lars Francke
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.5.0
>
> Attachments: Test_fix_for_HDFS-171811.patch
>
>
> When calling WebHDFS to create a file (I'm sure the same problem occurs for 
> other actions e.g. OPEN but I haven't checked all of them yet) it will 
> happily redirect to nodes that are in maintenance.
> The reason is in the 
> [{{chooseDatanode}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L307C9-L315]
>  method in {{NamenodeWebHdfsMethods}} where it will only call the 
> {{BlockPlacementPolicy}} (which considers all these edge cases) in case the 
> {{remoteAddr}} (i.e. the address making the request to WebHDFS) is also 
> running a DataNode.
>  
> In all other cases it just refers to 
> [{{NetworkTopology#chooseRandom}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L342-L343]
>  which does not consider any of these circumstances (e.g. load, maintenance).
> I don't understand the reason to not just always refer to the placement 
> policy and we're currently testing a patch to do just that.
> I have attached a draft patch for now.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2024-02-20 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1505/

[Feb 18, 2024, 1:15:40 AM] (github) HDFS-17146.Use the dfsadmin -reconfig 
command to initiate reconfiguration on all decommissioning datanodes. (#6504) 
Contributed by Hualong Zhang.
[Feb 19, 2024, 12:34:59 AM] (github) HDFS-17361. DiskBalancer: Query command 
support with multiple nodes (#6508)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17392) NameNode rolls frequently with "EC replicas to be deleted are not in the candidate" error

2024-02-20 Thread Rick Weber (Jira)

Rick Weber created HDFS-17392:
-

 Summary: NameNode rolls frequently with "EC replicas to be deleted 
are not in the candidate" error
 Key: HDFS-17392
 URL: https://issues.apache.org/jira/browse/HDFS-17392
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 3.3.6
Reporter: Rick Weber


Recently upgraded my clusters from Hadoop v3.3.4 to Hadoop v3.3.6 and noticed a 
lot of Namenode instability.  Basically after about 1 hour, the active namenode 
shuts down and the "next" one takes over.

Looking into the shutdown reasons, I'm seeing errors similar to
{code:java}
2024-02-20 12:05:37,352 INFO 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Rescan of 
postponedMisreplicatedBlocks completed in 8 msecs. 6639943 blocks are left. 1 
blocks were removed.
2024-02-20 12:05:37,352 ERROR 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: RedundancyMonitor 
thread received Runtime exception.
java.lang.IllegalArgumentException: The EC replicas to be deleted are not in 
the candidate list
at 
org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseExcessRedundancyStriped(BlockManager.java:4082)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseExcessRedundancies(BlockManager.java:3970)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processExtraRedundancyBlock(BlockManager.java:3957)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processMisReplicatedBlock(BlockManager.java:3898)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.rescanPostponedMisreplicatedBlocks(BlockManager.java:2898)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$RedundancyMonitor.run(BlockManager.java:5053)
at java.lang.Thread.run(Thread.java:750)
2024-02-20 12:05:37,357 INFO org.apache.hadoop.util.ExitUtil: Exiting with 
status 1: java.lang.IllegalArgumentException: The EC replicas to be deleted are 
not in the candidate list {code}
Looking through the code path itself, there is a check for 
`Preconditions.checkArgument()` to ensure that a given block chosen for 
deletion is actually one of the valid blocks.  If not, then the NN shuts down.

This is likely a symptom to a larger issue, such as how is a block being chosen 
that is not in the candidate list.

The remainder of the cluster has services such as SPS and Balancer service 
disabled, so that the only movement of data should be whatever is "organically" 
chosen by the NameNode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17391) Adjust the checkpoint io buffer size to the chunk size

2024-02-20 Thread lei w (Jira)

lei w created HDFS-17391:


 Summary: Adjust the checkpoint io buffer size to the chunk size
 Key: HDFS-17391
 URL: https://issues.apache.org/jira/browse/HDFS-17391
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: lei w


Adjust the checkpoint io buffer size to the chunk size to reduce checkpoint 
time 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2024-02-20 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.TestDFSInotifyEventInputStream 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.hdfs.TestFileCreation 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.TestLeaseRecovery2 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.TestFileLengthOnClusterRestart 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.yarn.sls.TestSLSRunner 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-compile-javac-root.txt
  [508K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-mvnsite-root.txt
  [592K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [244K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.8M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1308/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt
  [16K]

Re: Discussion about NameNode Fine-grained locking

2024-02-20 Thread Yuanbo Liu

Nice to see this feature brought up. We've implemented this feature
internally and gained significant performance improvement. I'll be glad to
work on some jiras if necessary.


On Tue, Feb 20, 2024 at 4:41 PM ZanderXu  wrote:

> Thank you everyone for reviewing this ticket.
>
> I think if there are no problems with the goal and the overall solution, we
> are ready to push this ticket forward and I will create some detailed
> sub-tasks for this ticket.
>
> I will split this project into three milestones to make this project
> cleaner for review and merge.
> Milestone 1: Replacing the current global lock with two locks, global FS
> lock and global BM lock. End-user can choose which locking mode to use
> through configuration.
> Milestone 2: Replacing the global FS write lock with directory tree-based
> fine-grained lock.
> Milestone 3: Replacing the global BM lock with directory tree-based
> fine-grained lock.
>
> Each milestone can be merged into the trunk branch in time, which has
> multiple benefits:
> 1. Phased performance improvements can be quickly used by everyone
> 2. All developers can better understand the implementation ideas of the
> fine-grained locking mechanism as soon as possible
> 3. Each milestone is developed based on the latest trunk branch to reduce
> conflicts
>
> If you have any concerns, please feel free to discuss them together.
> I hope you can join us to push this project forward together, thanks.
>
>
> On Mon, 5 Feb 2024 at 11:33, haiyang hu  wrote:
>
> > Thank you for raising the issue of this long-standing bottleneck, this
> > will be a very important improvement!
> >
> > Hopefully can participate and push forward together.
> >
> > Best Regards~
> >
> > Brahma Reddy Battula  于2024年2月3日周六 00:40写道：
> >
> >> Thanks for bringing this and considering all the history around this.
> >> One of the outstanding bottleneck(global lock) from a long time.
> >>
> >> Hopefully we can push forward this time.
> >>
> >>
> >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei  wrote:
> >>
> >> > Thanks for driving this. It's very meaningful. The performance
> >> improvement
> >> > looks very good.
> >> >
> >> > Many users are facing the write performance issue. As far as I know,
> >> some
> >> > companies already implemented the similar idea on their internal
> >> branches.
> >> > But the internal branch is very different from the community one. So
> >> it's
> >> > very hard to be in sync with the community. If this improvement can be
> >> > involved in the community, that would be great to both end-user and
> the
> >> > community.
> >> >
> >> > It is very worth doing.
> >> >
> >> > Zengqiang XU  于2024年2月2日周五 11:07写道：
> >> >
> >> > > Hi everyone
> >> > >
> >> > > I have started a discussion about NameNode Fine-grained Locking to
> >> > improve
> >> > > performance of write operations in NameNode.
> >> > >
> >> > > I started this discussion again for serval main reasons:
> >> > > 1. We have implemented it and gained nearly 7x performance
> >> improvement in
> >> > > our prod environment
> >> > > 2. Many other companies made similar improvements based on their
> >> internal
> >> > > branch.
> >> > > 3. This topic has been discussed for a long time, but still without
> >> any
> >> > > results.
> >> > >
> >> > > I hope we can push this important improvement in the community so
> that
> >> > all
> >> > > end-users can enjoy this significant improvement.
> >> > >
> >> > > I'd really appreciate you can join in and work with me to push this
> >> > feature
> >> > > forward.
> >> > >
> >> > > Thanks very much.
> >> > >
> >> > > Ticket: HDFS-17366 <
> https://issues.apache.org/jira/browse/HDFS-17366>
> >> > > Design: NameNode Fine-grained locking based on directory tree
> >> > > <
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing
> >> > > >
> >> > >
> >> >
> >>
> >
>

[jira] [Created] (HDFS-17390) [FGL] FSDirectory supports this fine-grained locking

2024-02-20 Thread ZanderXu (Jira)

ZanderXu created HDFS-17390:
---

 Summary: [FGL] FSDirectory supports this fine-grained locking
 Key: HDFS-17390
 URL: https://issues.apache.org/jira/browse/HDFS-17390
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: ZanderXu


{code:java}
/**
 * The directory lock dirLock provided redundant locking.
 * It has been used whenever namesystem.fsLock was used.
 * dirLock is now removed and utility methods to acquire and release dirLock
 * remain as placeholders only
 */
void readLock() {
  assert namesystem.hasReadLock() : "Should hold namesystem read lock";
}

void readUnlock() {
  assert namesystem.hasReadLock() : "Should hold namesystem read lock";
}

void writeLock() {
  assert namesystem.hasWriteLock() : "Should hold namesystem write lock";
}

void writeUnlock() {
  assert namesystem.hasWriteLock() : "Should hold namesystem write lock";
}

boolean hasWriteLock() {
  return namesystem.hasWriteLock();
}

boolean hasReadLock() {
  return namesystem.hasReadLock();
}

@Deprecated // dirLock is obsolete, use namesystem.fsLock instead
public int getReadHoldCount() {
  return namesystem.getReadHoldCount();
}

@Deprecated // dirLock is obsolete, use namesystem.fsLock instead
public int getWriteHoldCount() {
  return namesystem.getWriteHoldCount();
}

public int getListLimit() {
  return lsLimit;
} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Discussion about NameNode Fine-grained locking

2024-02-20 Thread ZanderXu

Thank you everyone for reviewing this ticket.

I think if there are no problems with the goal and the overall solution, we
are ready to push this ticket forward and I will create some detailed
sub-tasks for this ticket.

I will split this project into three milestones to make this project
cleaner for review and merge.
Milestone 1: Replacing the current global lock with two locks, global FS
lock and global BM lock. End-user can choose which locking mode to use
through configuration.
Milestone 2: Replacing the global FS write lock with directory tree-based
fine-grained lock.
Milestone 3: Replacing the global BM lock with directory tree-based
fine-grained lock.

Each milestone can be merged into the trunk branch in time, which has
multiple benefits:
1. Phased performance improvements can be quickly used by everyone
2. All developers can better understand the implementation ideas of the
fine-grained locking mechanism as soon as possible
3. Each milestone is developed based on the latest trunk branch to reduce
conflicts

If you have any concerns, please feel free to discuss them together.
I hope you can join us to push this project forward together, thanks.

On Mon, 5 Feb 2024 at 11:33, haiyang hu  wrote:

> Thank you for raising the issue of this long-standing bottleneck, this
> will be a very important improvement!
>
> Hopefully can participate and push forward together.
>
> Best Regards~
>
> Brahma Reddy Battula  于2024年2月3日周六 00:40写道：
>
>> Thanks for bringing this and considering all the history around this.
>> One of the outstanding bottleneck(global lock) from a long time.
>>
>> Hopefully we can push forward this time.
>>
>>
>> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei  wrote:
>>
>> > Thanks for driving this. It's very meaningful. The performance
>> improvement
>> > looks very good.
>> >
>> > Many users are facing the write performance issue. As far as I know,
>> some
>> > companies already implemented the similar idea on their internal
>> branches.
>> > But the internal branch is very different from the community one. So
>> it's
>> > very hard to be in sync with the community. If this improvement can be
>> > involved in the community, that would be great to both end-user and the
>> > community.
>> >
>> > It is very worth doing.
>> >
>> > Zengqiang XU  于2024年2月2日周五 11:07写道：
>> >
>> > > Hi everyone
>> > >
>> > > I have started a discussion about NameNode Fine-grained Locking to
>> > improve
>> > > performance of write operations in NameNode.
>> > >
>> > > I started this discussion again for serval main reasons:
>> > > 1. We have implemented it and gained nearly 7x performance
>> improvement in
>> > > our prod environment
>> > > 2. Many other companies made similar improvements based on their
>> internal
>> > > branch.
>> > > 3. This topic has been discussed for a long time, but still without
>> any
>> > > results.
>> > >
>> > > I hope we can push this important improvement in the community so that
>> > all
>> > > end-users can enjoy this significant improvement.
>> > >
>> > > I'd really appreciate you can join in and work with me to push this
>> > feature
>> > > forward.
>> > >
>> > > Thanks very much.
>> > >
>> > > Ticket: HDFS-17366 
>> > > Design: NameNode Fine-grained locking based on directory tree
>> > > <
>> > >
>> >
>> https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing
>> > > >
>> > >
>> >
>>
>

[jira] [Resolved] (HDFS-17181) WebHDFS not considering whether a DN is good when called from outside the cluster

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[jira] [Created] (HDFS-17392) NameNode rolls frequently with "EC replicas to be deleted are not in the candidate" error

[jira] [Created] (HDFS-17391) Adjust the checkpoint io buffer size to the chunk size

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

Re: Discussion about NameNode Fine-grained locking

[jira] [Created] (HDFS-17390) [FGL] FSDirectory supports this fine-grained locking

Re: Discussion about NameNode Fine-grained locking

8 matches

Site Navigation

Mail list logo

Footer information