Re: [DISCUSS] Future Release and Support Strategy for Hadoop

lost in the nested historical JIRAs for Java 17 >> > support [1]. I'd like to start by summarizing my understanding of work done >> > and work remaining for Java 17 as well as other significant features the >> > community wants to ship in 3.5.0: >> >> >

Re: [VOTE] Release Apache Hadoop 3.4.2

gt; > Thank you, > Ahmar > > > From: Masatake Iwasaki > Sent: Thursday, June 5, 2025 4:46:06 PM > To: Wei-Chiu Chuang; Ahmar Suhail > Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; > mapreduce-...@hadoop.apache.org; yarn-..

Re: [VOTE] Release Apache Hadoop 3.4.2

e Iwasaki Sent: Thursday, June 5, 2025 4:46:06 PM To: Wei-Chiu Chuang; Ahmar Suhail Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: RE: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.2 CAUTION: This email originated from

Re: [VOTE] Release Apache Hadoop 3.4.2

Hi Wei-Chiu Chuang, I really wish we can make ARM64 binaries. Would like to find time to work on the two jiras mentioned. I submitted PRs for YARN-11712 and YARN-11713. Could you check the patches? Thanks, Masatake Iwasaki On 2025/06/05 1:14, Wei-Chiu Chuang wrote: I really wish we can make

Re: [VOTE] Release Apache Hadoop 3.4.2

I really wish we can make ARM64 binaries. Would like to find time to work on the two jiras mentioned. On Wed, May 28, 2025 at 5:25 AM Ahmar Suhail wrote: > Hey all, > > The first release candidate for Hadoop 3.4.2 is now available for voting. > > There are a couple of things to note: > > 1/ No A

Re: [VOTE] Release Apache Hadoop 3.4.2

t;> will do the merge and test >>> >>> On Mon, 2 Jun 2025 at 16:51, Suhail, Ahmar >>> wrote: >>> >>>> Steve - I created a PR on hadoop-release-support with my properties >>>> file: https://github.com/apache/hadoop-relea

Re: [VOTE] Release Apache Hadoop 3.4.2

the documentation for the new >>> RC if it is created. >>> >>> >>> Mukund - I realised too late that I messed up the commit for that one. >>> Will figure out how to fix.. >>> >>> >>> From: Steve Loughran >

Re Hadoop 3.5.0/Java 17 Release Planning

made significant progress on the JUnit 5 upgrade [3], and we > have a list of several PRs remaining to be reviewed and committed [4]. > - There is some sun.misc usage that needs alternate implementations [5] > [6]. There are PRs, but they haven't had any recent activity. > -

Re: [VOTE] Release Apache Hadoop 3.4.2

___ >> From: Steve Loughran >> Sent: Monday, June 2, 2025 4:25:00 PM >> To: Ahmar Suhail >> Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; >> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org >> Subject: RE: [EXTERNAL] [VOTE] Release Apache

Re: [VOTE] Release Apache Hadoop 3.4.2

> From: Steve Loughran > Sent: Monday, June 2, 2025 4:25:00 PM > To: Ahmar Suhail > Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; > mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org > Subject: RE: [EXTERNAL] [VOTE] Release Apa

Re: [VOTE] Release Apache Hadoop 3.4.2

. Will figure out how to fix.. From: Steve Loughran Sent: Monday, June 2, 2025 4:25:00 PM To: Ahmar Suhail Cc: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: RE: [EXTERNAL] [VOTE

Re: [VOTE] Release Apache Hadoop 3.4.2

can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file On Wed, 28 May 2025 at 13:25, Ahmar Suhail wrote: > Hey all, > > The first release candidate for Ha

Re: [VOTE] Release Apache Hadoop 3.4.2

* git_jira_fix_version_check.py reported some inconsistencies between commits and JIRA. I updated JIRA's "Fix Versions/s" and "Resolution" along with the commit status. Since HADOOP-19580 was commited to branch-3.4.2 but not in the RC1, it will be contained if RC2 is created. "Fix Versions/s"

Re: [VOTE] Release Apache Hadoop 3.4.2

Thanks for driving the release, Ahmar. I found some issues on documentations. While it is not worth for sinking the RC1, it would be nice to fix if we cut RC2. * The top page of site documentation looks mentioning changes of 3.4.1. https://github.com/apache/hadoop/blob/release-3.4.2-RC1/hado

Re: [VOTE] Release Apache Hadoop 3.4.2

Thanks for starting this! I tried accessing the S3 bucket using the RC by dropping in the bundle jar and it works. Will test more! On Wed, May 28, 2025 at 7:25 AM Ahmar Suhail wrote: > Hey all, > > The first release candidate for Hadoop 3.4.2 is now available for voting. > > There are a couple o

Re: [VOTE] Release Apache Hadoop 3.4.2

Thanks Ahmar for your work. Will try to verify and give feedback here. Best Regards, - He Xiaoqiao On Wed, May 28, 2025 at 8:56 PM Ahmar Suhail wrote: > > Hey all, > > The first release candidate for Hadoop 3.4.2 is now available for voting. > > There are a couple of things to note: > > 1/ No Ar

Re: [DISCUSS] Optimize the invoke retry of async repsonder when some nameservices are slow

+ hdfs-dev. Thanks Haobo for your proposal. As you mentioned above, this may be related to RBF and ARR features, right? IMO, it is necessary to improve responder performance, but I am a little confused about nameservice will slow down the whole system, The first glance is client or package size sh

[jira] [Resolved] (HDFS-17752) Host2DatanodeMap will not update when re-register a node with a different hostname

ill not update when re-register a node with a different > hostname > -- > > Key: HDFS-17752 > URL: https://issues.apache.org/jira/browse/HDFS-17752 >

Re: Fw: Data copy from HDFS to MinIO regularly

distcp can upload a directory tree of changed files, for cloud storage it looks for the different in file timestamps Otherwise, the HDFS namenode has an log4j audit logger: org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit This prints out all namenode filesystem operations in a structured

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

-1 Signatures look good, but I found a few spots that need an update for the new protobuf version: https://issues.apache.org/jira/browse/HADOOP-19510 https://github.com/apache/hadoop-thirdparty/pull/44 Chris Nauroth On Tue, Mar 18, 2025 at 8:47 AM Wei-Chiu Chuang wrote: > I just did > gpg -

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

thanks. will kick off a new build. Did anyone else see problems with the gpg versions? I destroyed the entire .gpgconf dir on the host i was building on; did manual verify after importing KEYS, and all seemed good. On Thu, 20 Mar 2025 at 04:37, Chris Nauroth wrote: > This is committed to tru

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

The time is up, the votes are in and they are +1 (binding) Ayush Saxena (binding) Chris Nauroth (binding) Steve Loughran (binding) Wei-Chiu Chuang (binding) Xiaoqiao He (binding) = 0: none -1: none Accordingly, the RC is the release; I will do the next steps in the process ASAP Steve On Tue,

Re: JDK 17 (or beyond) upgrade

Hello Vincent, There isn't a firm timeline yet, but the community is rallying around a 3.5.0 release with full support for JDK 17. In addition to the issue you linked, there is also this one: https://issues.apache.org/jira/browse/HADOOP-17177 Known work related to JDK 17 compatibility is going t

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Update: +1(binding). On Tue, Mar 25, 2025 at 7:10 PM Steve Loughran wrote: > > > pushed up. thanks for spotting that. Added another 3p. target to the > release-support build file. Really trying to automate all these steps > > 3p.git-tag-push: > [git] To github.com:apache/hadoop-thirdparty.

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

pushed up. thanks for spotting that. Added another 3p. target to the release-support build file. Really trying to automate all these steps 3p.git-tag-push: [git] To github.com:apache/hadoop-thirdparty.git [git] * [new tag] release-1.4.0-RC1 -> release-1.4.0-RC1 On Mon, 24 M

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Thanks Steve for driving this release. +0 and will give my +1 while tag to be fixed. [Y] LICENSE files exist and NOTICE is included. [Y] Build the source code on Ubuntu and OpenJDK 11. [Y] Verified checksums and Signature. [Y] Skimmed over the contents of ReleaseNotes and ChangeLogs. [N] release

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

+1 (Binding) * Built from source * Verified checksums * Verified Signature * Verified no diff b/w branch-1.4 & src tar [1] * Skimmed over the contents of ReleaseNotes & ChangeLog * Checked the NOTICE & LICENSE files Thanx Steve for driving the release. Good Luck!!! -Ayush [1] I think releas

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

+1 * Verified signatures. * LICENSE-binary has the expected dependencies and versions. * Site looks good too. * Specifically confirmed inclusion of HADOOP-19510, previously identified during RC0. Thanks for the release, Steve! Chris Nauroth On Fri, Mar 21, 2025 at 10:43 AM Steve Loughran wrot

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I am +1 too. * Verified signature. * Release notes and changelogs look good. On Fri, Mar 21, 2025 at 4:35 PM Chris Nauroth wrote: > +1 > > * Verified signatures. > * LICENSE-binary has the expected dependencies and versions. > * Site looks good too. > * Specifically confirmed inclusion of HADOO

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I did try everything, no problem with the keys or so. -Ayush On Thu, 20 Mar 2025 at 23:21, Steve Loughran wrote: > thanks. will kick off a new build. > > Did anyone else see problems with the gpg versions? > > I destroyed the entire .gpgconf dir on the host i was building on; did > manual verif

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

This is committed to trunk and branch-1.4.0 now, ready for another RC. Thanks to Aayush for the code review. Chris Nauroth On Wed, Mar 19, 2025 at 1:51 PM Chris Nauroth wrote: > -1 > > Signatures look good, but I found a few spots that need an update for the > new protobuf version: > > https:/

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

I just did gpg --verify .tar.gz.asc .tar.gz Ok after importing hadoop KEYS , this is the output which looks good to me: gpg: Signature made Fri Mar 14 08:56:01 2025 PDT gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 gpg:

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Let me check that; getting signing set up on the transient vm was tricky as my yubikey one wasn't there so I used my other .. regarding the jersey -bigger piece off work to actually pick up, so not touching it. On Fri, 14 Mar 2025 at 21:46, Wei-Chiu Chuang wrote: > Looks like your GPG key expir

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

can you give me what command you ran to check the signature, and see if you can update all your keys from the central servers? My key thinks it is not expired, so maybe its the KEYS file i need to update or you need to refresh your keys. I will have a look at the keys file tomorrow On Fri, 14 Mar

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Verified SHA-512 checksum, GPG signature -- all good except that the signing key expired. Confirmed git tag release-1.4.0-RC0 and branch name branch-1.4.0 both maps to commit hash 5595f1357eaad02e1cfb660bcce7fd34515197ff. Confirmed it is released as 1.4.0. Noticed HADOOP-19075

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Looks like your GPG key expired? gpg: Signature made Fri Mar 14 08:56:00 2025 PDT gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 gpg: Good signature from "Steve Loughran (ASF code sign key - 2018) < ste...@apache.org>" [expired] gpg: aka "[jpeg image of

Re: Can not run HDFS balancer cause metrics already exists

try to delete /system/balancer.id and search some error or warn logs in namenode. Replied Message | From | Sébastien Rebecchi | | Date | 3/9/2025 23:08 | | To | Zhanghaobo | | Cc | hadoop-user-maillist, hdfs-dev | | Subject | Re: Can not run HDFS balancer cause metrics already exists

Re: Can not run HDFS balancer cause metrics already exists

got it, you can use it as a service and see what will happen. Replied Message | From | Sébastien Rebecchi | | Date | 03/09/2025 22:22 | | To | Zhanghaobo | | Cc | u...@hadoop.apache.org、hdfs-dev@hadoop.apache.org | | Subject | Re: Can not run HDFS balancer cause metrics already exists

Re: Can not run HDFS balancer cause metrics already exists

Hi, @Sébastien Rebecchi Don't know more details about how you start balancer, did you use -asService? Replied Message | From | Sébastien Rebecchi | | Date | 3/9/2025 18:03 | | To | , | | Subject | Re: Can not run HDFS balancer cause metrics already exists | Hello

[jira] [Created] (HDFS-17752) Host2DatanodeMap will not update when re-register a node with a different hostname

WenjingLiu created HDFS-17752: - Summary: Host2DatanodeMap will not update when re-register a node with a different hostname Key: HDFS-17752 URL: https://issues.apache.org/jira/browse/HDFS-17752 Project

Re: [DISCUSS] Hadoop 3.4.2 release

+1, thanx Ahmar for driving it. I plan to c-pick YARN-11738 to branch-3.4, if folks are ok with it, that is need by Tez, or atleast makes life easier there -Ayush On Sun, 2 Mar 2025 at 01:30, Chao Sun wrote: > +1, thanks Ahmar! > > On Fri, Feb 28, 2025 at 7:21 AM larry mccay wrote: > > > +1 >

Re: [DISCUSS] Hadoop 3.4.2 release

+1, thanks Ahmar! On Fri, Feb 28, 2025 at 7:21 AM larry mccay wrote: > +1 > > Thanks, Ahmar! > > On Fri, Feb 28, 2025 at 7:37 AM slfan1989 wrote: > > > +1 > > > > I can provide assistance with the ARM release, as I originally planned to > > release Hadoop 3.5.0, which supports JDK17, and have b

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thanks, Ahmar! On Fri, Feb 28, 2025 at 7:37 AM slfan1989 wrote: > +1 > > I can provide assistance with the ARM release, as I originally planned to > release Hadoop 3.5.0, which supports JDK17, and have been conducting > related tests. > > My approach is a workaround, and I hope it helps. > >

Re: [DISCUSS] Hadoop 3.4.2 release

+1 I can provide assistance with the ARM release, as I originally planned to release Hadoop 3.5.0, which supports JDK17, and have been conducting related tests. My approach is a workaround, and I hope it helps. Best Regards, - Shilun Fan On Thu, Feb 27, 2025 at 9:08 PM Bhavik Patel wrote: > H

Re: [DISCUSS] Hadoop 3.4.2 release

Hello PMCs, Can we do a release for hadoop-thirdparty 1.4.0 or 1.3.1 with the fix for CVE-2024-7254(HADOOP-19289) and bump hadoop-thirdparty version in hadoop-3.4.2? On Thu, 27 Feb 2025, 5:15 pm Anuj Modi, wrote: > +1 > Thanks for initiative Ahmar. > > For ABFS we would like to highlight two ma

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thanks for initiative Ahmar. For ABFS we would like to highlight two major changes in the upcoming release. 1. Announcing Deprecation of WASB driver: Jira , Documentation

Re: [DISCUSS] Hadoop 3.4.2 release

+1. Thanks Ahmar. On Thu, Feb 27, 2025 at 2:31 AM Chris Nauroth wrote: > +1 > > Thank you, Ahmar! > > Chris Nauroth > > > On Wed, Feb 26, 2025 at 8:31 AM Ahmar Suhail wrote: > > > Hey all, > > > > Would like to start a discussion for a Hadoop 3.4.2 release. > > > > For S3A, we would like to rel

Re: [DISCUSS] Hadoop 3.4.2 release

+1 Thank you, Ahmar! Chris Nauroth On Wed, Feb 26, 2025 at 8:31 AM Ahmar Suhail wrote: > Hey all, > > Would like to start a discussion for a Hadoop 3.4.2 release. > > For S3A, we would like to release support for conditional writes, and > integration of a new analytics input stream. > > I hav

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

; common-...@hadoop.apache.org ; Hui Fei < > feihui.u...@gmail.com>; hfutzhan...@163.com > *Subject:* Re: [VOTE] Request to merge branch HDFS-17531 into trunk. > > +1 > > Best Regards, > - Shilun Fan > > On Fri, Feb 14, 2025 at 3:22 PM jian zhang wrote: > > > Hi, al

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

; Hui Fei ; hfutzhan...@163.com Subject: Re: [VOTE] Request to merge branch HDFS-17531 into trunk. +1 Best Regards, - Shilun Fan On Fri, Feb 14, 2025 at 3:22 PM jian zhang wrote: > Hi, all, the development of the asynchronous router functionality has been > completed. Thanks to a

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

feature. >> >> >> >> Best Regards, >> >> Jian Zhang >> >> >> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >> >> >> >>> @Hui Fei Hi, Sir: >> >>> For the first opinion, I have create an umbrella JIRA >&

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

t point > does not exist, etc.), it directly puts the response into the response > queue; otherwise, it sends the request to the asynchronous handler thread > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately w

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

>> >> >>> >> Best Regards, >>> >> Jian Zhang >>> >> >>> >> Zhanghaobo 于2025年1月22日周三 18:22写道： >>> >> >>> >>> @Hui Fei Hi, Sir: >>> >>> For the first opinion, I have create an umbr

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

+1 - Zhang Haobo Replied Message | From | Xiaoqiao He | | Date | 02/14/2025 19:38 | | To | Ayush Saxena | | Cc | Hui Fei , jian zhang , Hdfs-dev , , | | Subject | Re: [VOTE] Request to merge branch HDFS-17531 into trunk. | +1 Best Regards, - He Xiaoqiao On Fri, Feb 14, 2025

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

equests from the request queue for preliminary > >> processing. If there are exceptions in the request (such as the mount > point > >> does not exist, etc.), it directly puts the response into the response > >> queue; otherwise, it sends the request to the asynchr

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

response into the response >> queue; otherwise, it sends the request to the asynchronous handler thread >> pool. >> - Async Handler: Puts the request into the call queue >> (connection.calls) of the connection thread and returns immediately without >> blocking and waiting.

Re: [VOTE] Request to merge branch HDFS-17531 into trunk.

queue; otherwise, it sends the request to the asynchronous handler thread > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately without > blocking and waiting. > - Async Responder: Is responsible for pr

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

t; Zhanghaobo 于2025年1月22日周三 18:22写道： >> >> >> >>> @Hui Fei Hi, Sir: >> >>> For the first opinion, I have create an umbrella JIRA >> >>> https://issues.apache.org/jira/browse/HDFS-17716 >> >>> and move non-core JIRA under i

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

16 > >>> and move non-core JIRA under it. > >>> > >>> Best Wishes > >>> Haobo Zhang > >>> > >>> Replied Message > >>> From Hui Fei > >>> Date 01/22/2025 17:37 > >>> To jian zhang > >>> Cc H

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

ve non-core JIRA under it. >>> >>> Best Wishes >>> Haobo Zhang >>> >>> Replied Message >>> From Hui Fei >>> Date 01/22/2025 17:37 >>> To jian zhang >>> Cc Hdfs-dev , >>> , >>>

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

For the first opinion, I have create an umbrella JIRA >> https://issues.apache.org/jira/browse/HDFS-17716 >> and move non-core JIRA under it. >> >> Best Wishes >> Haobo Zhang >> >> Replied Message >> From Hui Fei >> Date 01/22/

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

.org/jira/browse/HDFS-17716 > and move non-core JIRA under it. > > Best Wishes > Haobo Zhang > > Replied Message > From Hui Fei > Date 01/22/2025 17:37 > To jian zhang > Cc Hdfs-dev , > , > Xiaoqiao He , > > > Subject Re: [DISCUSS] Request

Re: [apache/hadoop] HDFS-17531. RBF: Aynchronous router RPC. (PR #7308)

Hi, slfan1989, Currently, this feature has been developed and all pipelines have passed. It is waiting for community voting before being merged into trunk. slfan1989 于2025年2月6日周四 16:55写道： > @KeeProMise @hfutatzhanghb > When is t

Re: Bug with ZStandardCodec in Hadoop,

Hello Sungwoo, It looks like HDFS-14099 was fixed in Apache Hadoop releases 3.2.3, 3.3.2 and 3.4.0. Users of Apache Hive 3.1.3 and earlier would be impacted. (Hive 3.1.3 used Hadoop 3.1.0.) However, Hive 4.0.0 used Hadoop 3.3.6, so I think we can consider this resolved by the latest Hive release.

Re: Bug with ZStandardCodec in Hadoop,

Thanks Sungwoo Park for sharing the details. I'm forwarding this to hdfs-dev@. I haven't had the chance to review the details in the ticket yet, but if you can reproduce the issue, I recommend creating an HDFS ticket and marking it as a blocker with the target versions set to the upcoming releases.

Re: Data not visible to new readers after SequenceFile.writer.hflush()

it's visible, but the namenode isn't updated so file.len() is still the same. if you go to the EOF and then try to read past it, you get the new data. I know, it's not "posix", but, well, view as eventual consistency on file length other thing to know: close() does an hflush, but not an hsync().

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Hello Thank you for answering. As I understood hflush must make the data visible to new readers but this is not the case actually for performance reasons. Can it be considered a bug? When you say 'the writer would need to record the client side visible length', I don't see anything to do that fr

Re: Data not visible to new readers after SequenceFile.writer.hflush()

This is Hadoop's unit test for sequence file exercising hsync/hflush: https://github.com/apache/hadoop/blob/61df1b27a797efd094328c7d9141b9e157e01bf4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java#L151 On Mon, Jan 27, 2025 at 9:53 AM Wei-Chiu Ch

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Do you have a unit test to reproduce? Note that for performance reasons, the actual HDFS hflush/hsync implementation does not update visible length at NameNode. The data is flushed to DataNode though. So the writer would need to record the client side visible length, and pass it to the reader, if

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Hi Sébastien, I replied to your same question on the hadoop-user@ thread. Let's keep the discussion there unless we discover there is some kind of HDFS bug to discuss. Thank you! Chris Nauroth On Mon, Jan 27, 2025 at 6:31 AM Sébastien Rebecchi wrote: > Hello, > > I got this issue using hadoop

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

, , Xiaoqiao He , | | Subject | Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk. | Got your idea. Thank you! - How about removing unfinished tasks and placing them under a new task as subtasks, like ARR improvements? If this feature is completed but there are still some open tasks, it

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

ervices are >>>> not fully utilized, leading to unreasonable resource allocation. >>>> >>>> III. Design and Improvements of the Asynchronous Router >>>> >>>> The asynchronous router solves the above problems by redesigning >>>

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

process and introducing an asynchronous processing >>> mechanism. Its core improvements include: >>> >>> - Handler: Retrieves requests from the request queue for preliminary >>> processing. If there are exceptions in the request (such as the mount point >&

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

nt point >> does not exist, etc.), it directly puts the response into the response >> queue; otherwise, it sends the request to the asynchronous handler thread >> pool. >> - Async Handler: Puts the request into the call queue >> (connection.calls) of the connec

Re: [DISCUSS] Request to merge branch HDFS-17531 into trunk.

d > pool. > - Async Handler: Puts the request into the call queue > (connection.calls) of the connection thread and returns immediately without > blocking and waiting. > - Async Responder: Is responsible for processing the responses > received by the connection thread. I

Re: [VOTE] Merge HDFS-17384 to trunk

t; > To ZanderXu >> > Cc Hdfs-dev , >> > , >> > Ayush Saxena , >> > Xiaoqiao He , >> > slfan1989 , >> > Zhanghaobo , >> > , >> > >> > >> > Subject Re: [VOTE] Merge HDFS-17384 to trunk >> > +1 (binding) >> > Revie

Re: [VOTE] Merge HDFS-17384 to trunk

> Ayush Saxena , > > Xiaoqiao He , > > slfan1989 , > > Zhanghaobo , > > , > > > > > > Subject Re: [VOTE] Merge HDFS-17384 to trunk > > +1 (binding) > > Reviewed sub PRs > > It has been running for a long time on prod clusters > > > &g

Re: [VOTE] Merge HDFS-17384 to trunk

Fei > Date 01/13/2025 17:14 > To ZanderXu > Cc Hdfs-dev , > , > Ayush Saxena , > Xiaoqiao He , > slfan1989 , > Zhanghaobo , > , > > > Subject Re: [VOTE] Merge HDFS-17384 to trunk > +1 (binding) > Reviewed sub PRs > It has been running for a

Re: [VOTE] Merge HDFS-17384 to trunk

+1 (binding) It has been running stably for several months on prod clusters Replied Message | From | Hui Fei | | Date | 01/13/2025 17:14 | | To | ZanderXu | | Cc | Hdfs-dev , , Ayush Saxena , Xiaoqiao He , slfan1989 , Zhanghaobo , , | | Subject | Re: [VOTE] Merge HDFS-17384 to trunk

Re: [VOTE] Merge HDFS-17384 to trunk

+1 (binding) Reviewed sub PRs It has been running for a long time on prod clusters ZanderXu 于2025年1月13日周一 16:35写道： > Hi all, > > I would like to propose that HDFS-17384 to be merged to trunk: > >- PR Link: PR#6762 - HDFS-17384: [FGL] HDFS NameNode FGL Phase I >

Re: Discussion about NameNode Fine-grained locking

A0%E6%B5%A9%E5%8D%9A&uid=hfutzhanghb%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc804eb39b0e7885aa8801c3bb66e497d.jpg&items=%5B%22hfutzhanghb%40163.com%22%5D> >> >> Replied Message ---- >> From haiyang hu >> Date 12/31/2024 23:08 >> To Ay

Re: Discussion about NameNode Fine-grained locking

> From haiyang hu > Date 12/31/2024 23:08 > To Ayush Saxena > Cc Hui Fei , > ZanderXu , > Hdfs-dev , > , > Xiaoqiao He , > slfan1989 , > > Subject Re: Discussion about NameNode Fine-grained locking > Thanks for your hard work and push it forward

Re: Discussion about NameNode Fine-grained locking

Thanks for your hard work and push it forward. It looks good, +1 for merging phase 1 codes, hope we can work together to promote this major HDFS optimization, so that more companies can benefit from it. Thanks everyone~ Ayush Saxena 于2024年12月31日周二 20:33写道： > +1, > Thanx folks for your efforts o

Re: Discussion about NameNode Fine-grained locking

+1, Thanx folks for your efforts on this! I didn't have time to review everything thoroughly, but my initial pass suggests it looks good or atleast is safe to merge. If I find some spare time, I'll test it further and submit a ticket or so if I encounter any issues. Good Luck!!! -Ayush On Tue, 3

Re: Discussion about NameNode Fine-grained locking

Thanks Zander for bringing this discussion again and trying your best to push it forward. It's really a long time since last discussion. It’s indeed time, +1 for merging phase 1 codes based on the following points - The phase 1 feature has been running at scale within companies for a long time -

Re: Discussion about NameNode Fine-grained locking

Hi, everyone: Time to Merge FGL Phase I The PR for *FGL Phase I* is ready for merging! Please take a moment to review and cast your vote: https://github.com/apache/hadoop/pull/6762. The *FGL Phase I* has been running successfully in production for over six months at *Shopee* and *BOSS Zhipin*, wi

Re: Compare metrics to detect performance issues in the latest versions

Thanks for your report. Change ML to hdfs-dev. Please first to choose common-dev, hdfs-dev or yarn-dev to discuss the issues. This ML is for security issues. Thanks. Best Regards, - He Xiaoqiao On Sun, Nov 10, 2024 at 2:26 AM 徐子康 wrote: > Hello! > > I am currently conducting a study to detect p

Re: [External Sender] Re: JDK17 support for Hadoop

; Asanuma , Hadoop Common , > yarn-dev , mapreduce-dev < > mapreduce-...@hadoop.apache.org>, Hdfs-dev > *Subject: *[External Sender] Re: JDK17 support for Hadoop > > Thank you all for your interest in this topic! > > In my opinion, upgrading is very beneficial. Based on

Re: [External Sender] Re: JDK17 support for Hadoop

, Hadoop Common , yarn-dev , mapreduce-dev , Hdfs-dev Subject: [External Sender] Re: JDK17 support for Hadoop Thank you all for your interest in this topic! In my opinion, upgrading is very beneficial. Based on the roadmap for Spark 4.0, we can see that JDK8/JDK11 is no longer the default version

Re: [INFO] branch HDFS-17531 rebase onto trunk

Hi Xiaoqiao He Thank you for taking the time to do this work! Best Regards, - Jian Zhang Xiaoqiao He 于2024年9月12日周四 15:54写道： > Hi All, > > Just rebase development feature branch HDFS-17531(RBF: Asynchronous router > RPC) onto trunk, > then keep branch HDFS-17531 closely with trunk. Thanks. > > B

Re: HDFS-17611. Move all DistCp execution logic to execute()

shadedclient error looks concerning. Please do not merge until you build it locally successfully without -DskipShade option On Thu, Sep 5, 2024 at 1:21 PM Steve Loughran wrote: > before I merge, has anyone here see any problems from this PR? > https://github.com/apache/hadoop/pull/7025 >

Re: Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (binding) * Successfully built from source with `mvn clean install` * Verified signatures and checksums Thanks Steve and Mukund for driving this release. Hope the release goes smoothly. Shilun Fan. On Fri, Aug 23, 2024 at 11:41 PM Shilun Fan wrote: +1 (Binding) > > * Built from source > *

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (Binding) * Built from source * Verified Signature(Signed by Mukund) * Verified Checksums * No diff b/w the git tag & src tar * Verified the NOTICE & LICENSE files * Skimmed over the contents of the site jar * Built Hadoop trunk with 1.3.0 version * Checked the hadoop test results here [1], not

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 (binding). Thanks Steve and Mukund for driving this release. Best Regards - He Xiaoqiao On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur wrote: > +1 ( binding) > > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran > > wrote: > > > Mukund and I have built a release candidate (RC1) for Had

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

de-protobuf) on > project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate > entry: META-INF/LICENSE.txt -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

--- > > mvn install package -DskipTests > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-protobuf) on > project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate > entry: META

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate entry: META-INF/LICENSE.txt -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For m

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

+1 ( binding) On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran wrote: > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty > 1.3.0. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/ > > The git tag is release-1.3.0-RC1, co

Re: [DISCUSS] what needs to be done to switch to java17

> One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5 Do you know any specific reason or docs mentioning that? Just curious as to why we need to migrate to JUnit 5 for JDK-17, Looking here [1], it seems like it supports JDK-21 as well and even [2] mentions like it does support

Re: [DISCUSS] what needs to be done to switch to java17

Thanks, everyone, for starting this discussion. This sounds like a good plan to start with. As I was working on the JUnit 4 to 5 upgrade, I paused for a while as I got occupied with other stuff. But I would be happy to complete it as part of the Java switch process. On Tue, Jul 30, 2024 at 4:21 AM

  1   2   3   4   5   6   7   8   9   10   >