[DISCUSS] lots of yetus failures in hdfs

[Steve Loughran]

PR's which trigger hdfs builds seem to hit a lot of hdfs test failures
https://github.com/apache/hadoop/pull/6675

Are these regressions or are the tests flaky?

I don't want commit patches which break things, yet hdfs tests seem
unreliable and so I'm dangerously tempted to +1 anyway...

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Steve Loughran]

On Thu, 30 May 2024 at 03:47, Xiaoqiao He  wrote:

> Strong +1. One concerns, how we define 'unnecessary branches', which
> mean how to distinguish the branches someone accidentally created, I try
> to traverse some of them but didn't get one obvious rule. Thanks.
>


If we take a snapshot into an archive repository then nothing will be lost
forever (actually, github fork should let us do this, so someone could just
log in as hadoop yetus and fork it there, perhaps)

Then we'd just look to see

   1. what feature branches are merged. e.g MR-379 is actually yarn itself.
   2. what feature branches are abandoned
   3. what release branches can we retire (all of 0.x, 1.x, early 2.x).
   maybe every branch not getting active maintenance, leaving only release
   tags?


Keeping all the tags will still result in a large repo. but removing
feature branches will potentially be good as the final merge was inevitably
a squashed merge...the intermediate chain of commits can be purged provided
there aren't tags associated with them




> Best Regards,
> - He Xiaoqiao
>
>
>
> On Wed, May 29, 2024 at 10:11 PM Ayush Saxena  wrote:
>
>> +1 for the proposal, the first thing might be to just drop the
>> unnecessary branches which are either from dependabot or someone
>> accidentally created a branch in the main repo rather than in their
>> fork, there are many, I don't think we need them in the archived repo
>> either
>>
>> Regarding (3) If you mean just branches, then should be ok, maybe lets
>> not touch the release tags for now IMO
>>
>> Regrading the regex, I tried this locally & it works
>> ```
>> ayushsaxena@ayushsaxena hadoop % git tag --delete  `git tag --list
>> 'ozone*'`
>> Deleted tag 'ozone-0.2.1-alpha-RC0' (was 90b070452bc7)
>> Deleted tag 'ozone-0.3.0-alpha' (was e9921ebf7e8d)
>> Deleted tag 'ozone-0.3.0-alpha-RC0' (was 3fbd1f15b894)
>> Deleted tag 'ozone-0.3.0-alpha-RC1' (was cdad29240e52)
>> Deleted tag 'ozone-0.4.0-alpha-RC0' (was 07fd26ef6d8c)
>> Deleted tag 'ozone-0.4.0-alpha-RC1' (was c4f9a20bbe55)
>> Deleted tag 'ozone-0.4.0-alpha-RC2' (was 6860c595ed19)
>> Deleted tag 'ozone-0.4.1-alpha' (was 687173ff4be4)
>> Deleted tag 'ozone-0.4.1-alpha-RC0' (was 9062dac447c8)
>> ayushsaxena@ayushsaxena hadoop %
>>
>> ```
>>
>> -Ayush
>>
>> On Wed, 29 May 2024 at 18:07, Steve Loughran
>>  wrote:
>> >
>> > I'm waiting for a git update of trunk to complete, not having done it
>> since
>> > last week. The 1.8 GB download is taking a long time over a VPN.
>> >
>> > Updating files: 100% (8518/8518), done.
>> > Switched to branch 'trunk'
>> > Your branch is up to date with 'apache/trunk'.
>> > remote: Enumerating objects: 4142992, done.
>> > remote: Counting objects: 100% (4142972/4142972), done.
>> > remote: Compressing objects: 100% (503038/503038), done.
>> > ^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
>> > remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
>> > Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
>> > Resolving deltas:  42% (1505182/3583765)
>> > ...
>> >
>> >
>> > We have too many branches and too many tags, which makes for big
>> downloads
>> > and slow clones, as well as complaints from git whenever I manually push
>> > things to gitbox.apache.org.
>> >
>> > I think we can/should clean up, which can be done as
>> >
>> >
>> >1. Create a hadoop-source-archive repository,
>> >2. into which we add all of the current hadoop-repository. This
>> ensures
>> >all the history is preserved.
>> >3. Delete all the old release branches, where old is defined as,
>> maybe <
>> >2.6?
>> >4. feature branches which are merged/abandoned
>> >5. all the single JIRA branches which are the same thing, "MR-279"
>> being
>> >a key example. ozone-* probably too.
>> >6. Do some tag pruning too. (Is there a way to do this with
>> wildcards? I
>> >could use it locally...)
>> >
>> > With an archive repo, all the old development history for branches off
>> the
>> > current release chain + tags are still available, but the core repo is
>> > much, much smaller.
>> >
>> > What do people think?
>> >
>> > If others are interested, I'll need some help carefully getting the
>> > hadoop-source-archive repo up. We'd need to somehow get all of hadoop
>> trunk
>> > into it.
>> >
>> > Meanwhile, I will cull some merged feature branches.
>> >
>> > Steve
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>

[DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Steve Loughran]

I'm waiting for a git update of trunk to complete, not having done it since
last week. The 1.8 GB download is taking a long time over a VPN.

Updating files: 100% (8518/8518), done.
Switched to branch 'trunk'
Your branch is up to date with 'apache/trunk'.
remote: Enumerating objects: 4142992, done.
remote: Counting objects: 100% (4142972/4142972), done.
remote: Compressing objects: 100% (503038/503038), done.
^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
Resolving deltas:  42% (1505182/3583765)
...


We have too many branches and too many tags, which makes for big downloads
and slow clones, as well as complaints from git whenever I manually push
things to gitbox.apache.org.

I think we can/should clean up, which can be done as


   1. Create a hadoop-source-archive repository,
   2. into which we add all of the current hadoop-repository. This ensures
   all the history is preserved.
   3. Delete all the old release branches, where old is defined as, maybe <
   2.6?
   4. feature branches which are merged/abandoned
   5. all the single JIRA branches which are the same thing, "MR-279" being
   a key example. ozone-* probably too.
   6. Do some tag pruning too. (Is there a way to do this with wildcards? I
   could use it locally...)

With an archive repo, all the old development history for branches off the
current release chain + tags are still available, but the core repo is
much, much smaller.

What do people think?

If others are interested, I'll need some help carefully getting the
hadoop-source-archive repo up. We'd need to somehow get all of hadoop trunk
into it.

Meanwhile, I will cull some merged feature branches.

Steve

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

followup: overnight work happy too.

one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing

> bin/hadoop checknative

2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop:  true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3

Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.

Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.

steve

On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:

> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>  > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > 

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

+1 binding

(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:

pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)

* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)

spark build is still ongoing, but I'm not going to wait. It is building,
which is key.

The core changes I needed in are at the dependency level and I've
verified they are good.

Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.

So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2





On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Steve Loughran]

 +1 for cutting hbase 1; it only reduces dependency pain (no more protobuf
2.5!)

Created the JIRA on that a few days back
https://issues.apache.org/jira/browse/YARN-11658

On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault 
wrote:

> Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
> you are at it you should probably update the hbase2 version, because 2.2.x
> is also very old and EOL. 2.5.x is the currently maintained release for
> hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
> well.
>
> On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena  wrote:
>
> > Hi Folks,
> > As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
> > for v2(2.2.4). The versions are specified over here: [1], how to build is
> > mentioned over here: [2]
> >
> > As of now we by default run our Jenkins "only" for HBase v1, so we have
> > seen HBase v2 profile silently breaking a couple of times.
> >
> > Considering there are stable versions for HBase v2 as per [3] & HBase v2
> > seems not too new, I have some suggestions, we can consider:
> >
> > * Make HBase v2 profile as the default profile & let HBase v1 profile
> stay
> > in our code.
> > * Ditch HBase v1 profile & just lets support HBase v2 profile.
> > * Let everything stay as is, just add a Jenkins job/ Github action which
> > compiles HBase v2 as well, so we make sure no change breaks it.
> >
> > Personally I would go with the second option, the last HBase v1 release
> > seems to be 2 years back, it might be pulling in some
> > problematic transitive dependencies & it will open scope for us to
> support
> > HBase 3.x when they have a stable release in future.
> >
> >
> > Let me know your thoughts!!!
> >
> > -Ayush
> >
> >
> > [1]
> >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
> >
> > [2]
> >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
> >
> > [3] https://hbase.apache.org/downloads.html
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[Steve Loughran]

I have been testing this all week, and a -1 until some very minor changes
go in.


   1. build the arm64 binaries with the same jar artifacts as the x86 one
   2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j.
   3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive
   dependencies


For #1 we have automation there in my client-validator module, which I have
moved to be a hadoop-managed project and tried to make more
manageable
https://github.com/apache/hadoop-release-support

This contains an ant project to perform a lot of the documented build
stages, including using SCP to copy down an x86 release tarball and make a
signed copy of this containing (locally built) arm artifacts.

Although that only works with my development environment (macbook m1 laptop
and remote ec2 server), it should be straightforward to make it more
flexible.

It also includes and tests a maven project which imports many of the
hadoop-* pom files and run some test with it; this caught some problems
with exported slf4j and log4j2 artifacts getting into the classpath. That
is: hadoop-common pulling in log4j 1.2 and 2.x bindings.

HADOOP-19084 fixes this; the build file now includes a target to scan the
dependencies and fail if "forbidden" artifacts are found. I have not been
able to stop logback ending on the transitive dependency list, but at least
there is only one slf4j there.

HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j
while the move to v2 is still something we have to consider a WiP.

I have tried doing some other changes to the packaging this week
- creating a lean distro without the AWS SDK
- trying to get protobuf-2.5 out of yarn-api
However, I think it is too late to try applying patches this risky.

I Believe we should get the 3.4.0 release out for people to start playing
with while we rapidly iterate 3.4.1 release out with
- updated dependencies (where possible)
- separate "lean" and "full" installations, where "full" includes all the
cloud connectors and their dependencies; the default is lean and doesn't.
That will cut the default download size in half.
- critical issues which people who use the 3.4.0 release raise with us.

That is: a packaging and bugs release, with a minimal number of new
features.

I've created HADOOP-19087
<https://issues.apache.org/jira/browse/HADOOP-19087> to cover this,
I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have put
a lot of work on 3.4.0 and probably need other people to take up the work
for next release. Who else is willing to participate? (Yes Mukund, I have
you in mind too)

One thing I would like to visit is: what hadoop-tools modules can we cut?
Are rumen and hadoop-streaming being actively used? Or can we consider them
implicitly EOL and strip. Just think of the maintenance effort we would
save.

---

Incidentally, I have tested the arm stuff on my raspberry pi5 which is now
running 64 bit linux. I believe it is the first time we have qualified a
Hadoop release with the media player under someone's television.

On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur 
wrote:

> Thanks, Shilun for putting this together.
>
> Tried the below things and everything worked for me.
>
> validated checksum and gpg signature.
> compiled from source.
> Ran AWS integration tests.
> untar the binaries and able to access objects in S3 via hadoop fs commands.
> compiled gcs-connector successfully using the 3.4.0 version.
>
> qq: what is the difference between RC1 and RC2? apart from some extra
> patches.
>
>
>
> On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:
>
>> Thank you for explaining this part!
>>
>> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to
>> generate
>> the ARM tar package, which should meet expectations.
>>
>> We also look forward to other members helping to verify.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
>> wrote:
>>
>> >
>> >
>> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
>> >
>> >>
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote.
>> >
>> >
>> >

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[Steve Loughran]

On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:

>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote.



that's exactly what the "arm.release" target in my client-validator does.
builds an arm tar with the x86 binaries but the arm native libs, signs it.



> Even automating that would be risky.
>
>
automating is the *only* way to do it; apache ant has everything needed for
this including the ability to run gpg.

we did this on the relevant 3.3.x releases and nobody has yet complained...

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)

[Steve Loughran]

so it'll have the s3a checksum and region stuff? that'd be wonderful! I've
been assuming i'd have to push out a 3.4.1 release for that alone.

On Sat, 10 Feb 2024 at 23:36, slfan1989  wrote:

> We will end the voting for hadoop-3.4.0-RC1 and will open the voting for
> hadoop-3.4.0-RC2 in 1-2 days.
>
> * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly
> submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0.
> * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Jan 29, 2024 at 10:33 PM slfan1989  wrote:
>
> > Apache Hadoop 3.4.0
> >
> > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/
> >
> > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1395/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from RC0
> >
> > * We've improved Hadoop 3.4.0 Highlight big features and improvements.
> > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules.
> > * Use validate-hadoop-client-artifacts[1] for packaging and verification.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] validate-hadoop-client-artifacts:
> > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > Thanks to steve for providing validate-hadoop-client-artifacts.
> >
> > Best Regards,
> > Shilun Fan.
> >
>

Re: Fw:Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Steve Loughran]

-1 I'm afraid, just due to staging/packaging issues.

This took me a few goes to get right myself, so nothing unusual.

Note I used my validator project which is set to retrieve binaries, check
signatures, run maven builds against staged artifacts *and clean up any
local copies first*and more.

This uses apache ant to manage all this:

https://github.com/steveloughran/validate-hadoop-client-artifacts

Here's the initial build.properties:file I used to try and manage this

## build.properties:
hadoop.version=3.4.0
rc=RC0
amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/
http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64

release=hadoop-${hadoop.version}-RC0
rc.dirname=${release}
release.native.binaries=false
git.commit.id=cdb8af4f22ec
nexus.staging.url=
https://repository.apache.org/content/repositories/orgapachehadoop-1391/
hadoop.source.dir=${local.dev.dir}/hadoop-trunk
##

When I did my own builds, all the artifacts created were without the RC0
suffix. It is critical this happens because the .sha512 checksums include
that in their paths

> cat hadoop-3.4.0-RC0.tar.gz.sha512
SHA512 (hadoop-3.4.0-RC0.tar.gz) =
e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab


Maven artifacts: staging problems

Couldn't build with a -Pstaging profile as the staging repository wasn't
yet closed -I tried to do that myself.

This failed with some rule problem

Event: Failed: Checksum Validation
Monday, January 15, 2024 14:37:13 GMT (GMT+)
typeId checksum-staging
failureMessage INVALID SHA-1:
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1'
failureMessage Requires one-of SHA-1:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1,
SHA-256:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256,
SHA-512:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512

I don't know precisely what this means...my guess is that the upload didn't
include everything.

Note my client-validator module can check this; just run its maven test
commands

mvn clean test -U -P3.4 -Pstaging

GPG signing: all good.

Picked your key up from the site ( ant gpg.keys ) ... first validation with
ant gpg.verify was unhappy as your key wasn't trusted. I've signed it and
pushed that signature up, so people who trust me get some reassurance about
you.

My build then failed as the gpg code couldn't find the
hadoop-3.4.0-aarch64.tar.gz.asc

The problem here is that although we want separate arm and x86 tar files,
we don't really want separate binaries as it only creates different jars in
the wild.

The way I addressed that was after creating that x86 release on an ec2 vm
and downloading it, I then did a local arm64 build and then created an arm
.tar.gz file, copied it into the same dir as the amd66 binaries but with
the arm64 .tar.gz filename, .asc and .sha512 checksum files all renamed
(checksum file patches to match the name).

https://github.com/steveloughran/validate-hadoop-client-artifacts?tab=readme-ov-file#arm64-binaries

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Steve Loughran]

wonderful! I'll be testing over the weekend

Meanwhile, new changes I'm putting in to trunk are tagged as fixed in 3.5.0
-correct?

steve


On Thu, 11 Jan 2024 at 05:15, slfan1989  wrote:

> Hello all,
>
> We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
> hadoop 3.4.0-RC version.
>
> The RC is available at:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
>
> This is a relatively big release (by Hadoop standard) containing about 2852
> commits.
>
> Please give it a try, this RC vote will run for 7 days.
>
> Feature highlights:
>
> DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
> 
> [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
> FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
> FsDatasetImpl datasetLock
> When there are many namespaces in a large cluster.
>
> YARN Federation improvements
> 
> [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
> improvements, including the following:
>
> 1. YARN Router now boasts a full implementation of all relevant interfaces
> including the ApplicationClientProtocol,
> ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
> 2. Enhanced support for Application cleanup and automatic offline
> mechanisms for SubCluster are now facilitated by the YARN Router.
> 3. Code optimization for Router and AMRMProxy was undertaken, coupled with
> improvements to previously pending functionalities.
> 4. Audit logs and Metrics for Router received upgrades.
> 5. A boost in cluster security features was achieved, with the inclusion of
> Kerberos support.
> 6. The page function of the router has been enhanced.
>
> Upgrade AWS SDK to V2
> 
> [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
> The S3A connector now uses the V2 AWS SDK.  This is a significant change at
> the source code level.
> Any applications using the internal extension/override points in the
> filesystem connector are likely to break.
> Consult the document aws\_sdk\_upgrade for the full details.
>
> hadoop-thirdparty will also provide the new RC0 soon.
>
> Best Regards,
> Shilun Fan.
>

Re: [DISCUSS][HDFS] Add rust binding for libhdfs

[Steve Loughran]

i hear owen o'malley has been learning rust, and as he left cloudera a year
ago, he'll be missing github and JIRA

On Thu, 21 Dec 2023 at 15:00, Ayush Saxena  wrote:

> It looks pretty challenging to me. Most of the committers aren't
> technically equipped to review this code, so getting the initial code
> reviewed & merged itself would be a challenge, as none of us can
> actually review the code.
>
> Looking at the repo, it has only 1 or 2 major contributors, which
> itself is a red flag, the bus factor is pretty low, if we don't find
> volunteers in future, we would be stuck with some dead code, which
> most of us don't know how to fix or maintain. If there is any CVE
> reported from this code post release, that would be a challenge for us
> to fix
>
> Quoting:
> > the Rust
> community has developed around 10 different HDFS client projects.
> However, almost all of them
> are no longer maintained.
>
> If they couldn't do, how we will be able to do that? and this isn't a
> very good statistic to quote :-)
>
>
> Well, I don't have objections on having this as a separate repo in
> Hadoop, if others are fine with it, I can try to help whatever is in
> my capacity, but I still have doubts on how easy would it be to push
> code or get votes on release of this project, which most of the people
> doesn't have knowledge & developing a community and stuff seems like a
> incubator thing to me.
>
> -Ayush
>
> On Thu, 21 Dec 2023 at 19:01, Xuanwo  wrote:
> >
> > Thanks Xiaoqiao He!
> >
> > Let me provide more context about this project.
> >
> > libhdfs-rust aims to provide native HDFS client support for Rust, a
> rapidly growing systems
> > programming language commonly used in modern infrastructure such as
> databases. With
> > libhdfs-rust, Rust developers can more easily integrate with HDFS.
> libhdfs-rust is analogous
> > to both libhdfs (C API) and libhdfspp <
> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp>
> (C++ API). Its current codebase builds upon libhdfs, but
> > there are plans to rewrite it entirely in pure rust. Consequently,
> libhdfs-rust will interface
> > directly with the HDFS Java client via JNI, making it fully parallel to
> both libhdfs and libhdfs-cpp.
> >
> > There are three possible ways for us to take:
> >
> > We have three options to consider:
> >
> > A: Integrate libhdfs-rust into the Hadoop repository, placing it under
> > 'hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native'.
> > B: Accept libhdfs-rust as a subproject and establish a new repository
> > named 'hadoop-hdfs-rust-client' (or another suitable name).
> > C: Maintain libhdfs-rust as an independent project outside of Hadoop.
> >
> > I personally prefer Option B since:
> >
> > For Option A
> >
> > The release process for Hadoop is already quite complex. We should avoid
> placing additional
> > burdens on the Release Managers, especially when it involves integrating
> a new language.
> >
> > And it's impossible to wait for libhdfs-rust mature and stable enough to
> catch up the release train.
> >
> > For Option C
> >
> > libhdfs-rust is exactly the same with libhdfs & libhdfspp <
> https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp>
> but for rust. Building a community for
> > libhdfs-rust outside of Hadoop is challenging. In fact, numerous
> attempts have been made: the Rust
> > community has developed around 10 different HDFS client projects.
> However, almost all of them
> > are no longer maintained.
> >
> > In conclusion, I believe that Option B is the best choice for us: we can
> develop a rust project in hadoop
> > community, attract more rust users, and recruit additional committers
> from the rust community.
> >
> >
> > On Wed, Dec 20, 2023, at 21:53, Xiaoqiao He wrote:
> > > Thanks Xuanwo for your work. I believe it is valuable to enlarge
> hadoop ecosystem.
> > >
> > > I am also concerned that it will involve more hard work to release and
> version match,
> > > especially for one who is not familiar with C or Rust.
> > > Moreover, I am not aware the difference between `accept hdfs-sys as
> part of hadoop
> > > project` and `one separate project`.
> > >
> > > I think one smooth solution is reference hadoop-thirdparty[1] which is
> one hadoop
> > > sub-project but split to separate repo and release line etc, if it is
> accepted.
> > >
> > > cc @Ayush Saxena  @Wei-Chiu Chuang  weic...@apache.org> @Iñigo Goiri  @Shilun Fan
>  and other folks, what
> > > do you think? Thanks.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > [1] https://github.com/apache/hadoop-thirdparty
> > >
> > > On Wed, Dec 20, 2023 at 6:17 PM Xuanwo  wrote:
> > >> I'm fine to start work under a new repo, and I'm willing to help
> maintain this repo. The repo could name after hadoop-libhdfs-rust or just
> 

Re: [VOTE] Hadoop 3.2.x EOL

[Steve Loughran]

+1

On Wed, 6 Dec 2023 at 04:09, Xiaoqiao He  wrote:

> Dear Hadoop devs,
>
> Given the feedback from the discussion thread [1], I'd like to start
> an official thread for the community to vote on release line 3.2 EOL.
>
> It will include,
> a. An official announcement informs no further regular Hadoop 3.2.x
> releases.
> b. Issues which target 3.2.5 will not be fixed.
>
> This vote will run for 7 days and conclude by Dec 13, 2023.
>
> I’ll start with my +1.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://lists.apache.org/thread/bbf546c6jz0og3xcl9l3qfjo93b65szr
>

Re: [DISCUSS] Make some release lines EOL

[Steve Loughran]

+1 for making 3.3 and 3.4 the maintained lines

3.2.x we should say -as it is true- that the age of the dependencies is
such that it is transitively insecure. To fix those, people must upgrade.

For 2.10.x, we should think about whether to cherrypick our own CVEs there,
but not actually do any new ASF releases.
I couldn't even get hold of a java7 JDK to do the release even if I wanted
to -the same must hold for many others; getting a new release qualified
would be hard. Best to say "upgrade time'.


This goes well with a 3.4.0 release, as there's a clear story: we have a
new 3.4.x line stabilising, if you want something already stable move onto
3.3.x if you hadn't already





On Mon, 4 Dec 2023 at 12:39, Xiaoqiao He  wrote:

> Hi folks,
>
> There are many discussions about which release lines should we still
> consider actively
> maintained in history. I want to launch this topic again, and try to get a
> consensus.
>
> From download page[1] and active branches page[2], we have the following
> release lines:
> Hadoop 3.3 Release (release-3.3.5 at Jun 22 2022),  360 commits checked in
> since last release.
> Hadoop 3.2 Release (release-3.2.4 at Jul 11, 2022) 36 commits checked in
> since last release.
> Hadoop 2.10 Release (release-2.10.2 at May 17, 2022) 24 commits checked in
> since last release.
>
> And Hadoop 3.4.0 will be coming soon which Shilun Fan (maybe cooperating
> with Ahmar Suhail?)
> has been actively working on getting the 3.4.0 release out.
>
> Considering the less updates for some active branches, should we declare to
> our downstream
> users that some of these lines will EOL?
>
> IMO we should announce EOL branch-2.10 and branch-3.2 which are not active
> now.
> Then we could focus on minor active branches (branch-3.3 and branch-3.4)
> and increase release pace.
>
> So how about to keep branch-3.3 and branch-3.4 release lines as actively
> maintained, And mark branch-2.10 and branch-3.2 EOL? Any opinions? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://hadoop.apache.org/releases.html
> [2]
>
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
>

[jira] [Resolved] (HDFS-17249) Fix TestDFSUtil.testIsValidName() unit test failure

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-17249.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Fix TestDFSUtil.testIsValidName() unit test failure
> ---
>
> Key: HDFS-17249
> URL: https://issues.apache.org/jira/browse/HDFS-17249
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: liuguanghua
>Assignee: liuguanghua
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> TestDFSUtil.testIsValidName runs failed when  
> assertFalse(DFSUtil.isValidName("/foo/:/bar")) , fixed it. 
> Add test case in TestDFSUtil.testIsValidName.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16791.
---
Resolution: Fixed

> Add getEnclosingRoot() API to filesystem interface and implementations
> --
>
> Key: HDFS-16791
> URL: https://issues.apache.org/jira/browse/HDFS-16791
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: fs
>Affects Versions: 3.3.6
>Reporter: Tom McCormick
>Assignee: Tom McCormick
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> At LinkedIn we run many HDFS volumes that are federated by either 
> ViewFilesystem or Router Based Federation. As our number of hdfs volumes 
> grows, we have a growing need to migrate data seemlessly across volumes.
> Many frameworks have a notion of staging or temp directories, but those 
> directories often live in random locations. We want an API getEnclosingRoot, 
> which provides the root path a file or dataset. 
> In ViewFilesystem / Router Based Federation, the enclosingRoot will be the 
> mount point.
> We will also take into account other restrictions for renames like 
> encryptions zones.
> If there are several paths (a mount point and an encryption zone), we will 
> return the longer path



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Fwd: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE

[Steve Loughran]

Can anyone from yarn or hdfs review this before i just upvote it hoping all
will be well. thanks

-- Forwarded message -
From: PJ Fanning (Jira) 
Date: Wed, 11 Oct 2023 at 20:27
Subject: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE
To: 


PJ Fanning created HADOOP-18933:
---

 Summary: upgrade netty to 4.1.100 due to CVE
 Key: HADOOP-18933
 URL: https://issues.apache.org/jira/browse/HADOOP-18933
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: PJ Fanning


follow up to https://issues.apache.org/jira/browse/HADOOP-18783

https://netty.io/news/2023/10/10/4-1-100-Final.html





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing

[Steve Loughran (Jira)]

Steve Loughran created HDFS-17224:
-

 Summary: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands 
failing
 Key: HDFS-17224
 URL: https://issues.apache.org/jira/browse/HDFS-17224
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: dfsadmin, test
Affects Versions: 3.4.0
Reporter: Steve Loughran


TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the 
static mbean isn't null. This is inevitably related to the fact that in test 
runs, the jvm is reused and so the mbean may be present from a previous test 
-maybe one which didn't clean up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Fwd: [jira] [Created] (HADOOP-18924) upgrade grpc jars to v1.53.0 due to CVEs

[Steve Loughran]

can we get some reviews from users of grpc/netty code here. otherwise i'd
upvoting without knowing what will break. thanks

-- Forwarded message -
From: PJ Fanning (Jira) 
Date: Mon, 9 Oct 2023 at 11:46
Subject: [jira] [Created] (HADOOP-18924) upgrade grpc jars to v1.53.0 due
to CVEs
To: 


PJ Fanning created HADOOP-18924:
---

 Summary: upgrade grpc jars to v1.53.0 due to CVEs
 Key: HADOOP-18924
 URL: https://issues.apache.org/jira/browse/HADOOP-18924
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Reporter: PJ Fanning


https://mvnrepository.com/artifact/io.grpc/grpc-protobuf



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17202) TestDFSAdmin.testAllDatanodesReconfig assertion failing (again)

[Steve Loughran (Jira)]

Steve Loughran created HDFS-17202:
-

 Summary: TestDFSAdmin.testAllDatanodesReconfig assertion failing 
(again)
 Key: HDFS-17202
 URL: https://issues.apache.org/jira/browse/HDFS-17202
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: dfsadmin, test
Affects Versions: 3.3.9
Reporter: Steve Loughran


surfacing in the test run for HADOOP-18895 pr 
https://github.com/apache/hadoop/pull/6073

```
Expecting:
 <["started at Thu Sep 14 23:14:07 GMT 2023SUCCESS: Changed property 
dfs.datanode.peer.stats.enabled",
" From: "false"",
" To: "true"",
" and finished at Thu Sep 14 23:14:07 GMT 2023."]>
to contain subsequence:
 <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled",
" From: "false"",
" To: "true""]>
```
looks like some logging race condition again as the "started at" output is on 
the same line as "SUCCESS". maybe something needs to add a \n after the started 
message, or before SUCCESS>



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Signing releases using automated release infra

[Steve Loughran]

could be good.

why not set it up for the third-party module first to see how well it works?

On Tue, 18 Jul 2023 at 21:05, Ayush Saxena  wrote:

> Something we can explore as well!!
>
> -Ayush
>
> Begin forwarded message:
>
> > From: Volkan Yazıcı 
> > Date: 19 July 2023 at 1:24:49 AM IST
> > To: d...@community.apache.org
> > Subject: Signing releases using automated release infra
> > Reply-To: d...@community.apache.org
> >
> > Abstract: Signing release artifacts using an automated release
> > infrastructure has been officially approved by LEGAL. This enables
> > projects to sign artifacts using, say, GitHub Actions.
> >
> > I have been trying to overhaul the Log4j release process and make it
> > as frictionless as possible since last year. As a part of that effort,
> > I wanted to sign artifacts in CI during deployment and in a
> > `members@a.o` thread[0] I explained how one can do that securely with
> > the help of Infra. That was in December 2022. It has been a long,
> > rough journey, but we succeeded. In this PR[1], Legal has updated the
> > release policy to reflect that this process is officially allowed.
> > Further, Infra put together guides[2][3] to assist projects. Logging
> > Services PMC has already successfully performed 4 Log4j Tools releases
> > using this approach, see its release process[4] for a demonstration.
> >
> > [0] (members only!)
> > https://lists.apache.org/thread/1o12mkjrhyl45f9pof94pskg55vhs61n
> > [1] https://github.com/apache/www-site/pull/235
> > [2] https://infra.apache.org/release-publishing.html#signing
> > [3]
> https://infra.apache.org/release-signing.html#automated-release-signing
> > [4]
> https://github.com/apache/logging-log4j-tools/blob/master/RELEASING.adoc
> >
> > # F.A.Q.
> >
> > ## Why shall a project be interested in this?
> >
> > It greatly simplifies the release process. See Log4j Tools release
> > process[4], probably the simplest among all Java-based ASF projects.
> >
> > ## How can a project get started?
> >
> > 1. Make sure your project builds are reproducible (otherwise there is
> > no way PMC can verify the integrity of CI-produced and -signed
> > artifacts)
> > 2. Clone and adapt INFRA-23996 (GPG keys in GitHub secrets)
> > 3. Clone and adapt INFRA-23974 (Nexus creds. in GitHub secrets for
> > snapshot deployments)
> > 4. Clone and adapt INFRA-24051 (Nexus creds. in GitHub secrets for
> > staging deployments)
> >
> > You might also want to check this[5] GitHub Action workflow for
> inspiration.
> >
> > [5]
> https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml
> >
> > ## Does the "automated release infrastructure" (CI) perform the full
> release?
> >
> > No. CI *only* uploads signed artifacts to Nexus. The release manager
> > (RM) still needs to copy the CI-generated files to SVN, PMC needs to
> > vote, and, upon consensus, RM needs to "close" the release in Nexus
> > and so on.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Steve Loughran]

1. we should patch gcs, to ignore those new keys.
2. i may be able to validate RC1 this weekend



On Fri, 16 Jun 2023 at 03:20, Wei-Chiu Chuang 
wrote:

> Overall so far so good.
>
> hadoop-api-shim:
> built, tested successfully.
>
> cloudstore:
> built successfully.
>
> Spark:
> built successfully. Passed hadoop-cloud tests.
>
> Ozone:
> One test failure due to unrelated Ozone issue. This test is being disabled
> in the latest Ozone code.
>
> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> to load library ozone_rocksdb_tools from both java.library.path &
> resource file libozone_rocksdb_t
> ools.so from jar.
> at
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
>
>
> Google gcs:
> There are two test failures. The tests were added recently by HADOOP-18724
>  in Hadoop 3.3.6. This
> is okay. Not production code problem. Can be addressed in GCS code.
>
> [ERROR] Errors:
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
> ucket/contract-test/testOpenFileApplyAsyncRead "fs.option.openfile.length"
>
>
>
>
>
> On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
> wrote:
>
> > The hbase-filesystem tests passed after reverting HADOOP-18596
> >  and HADOOP-18633
> >  from my local tree.
> > So I think it's a matter of the default behavior being changed. It's not
> > the end of the world. I think we can address it by adding an incompatible
> > change flag and a release note.
> >
> > On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
> > wrote:
> >
> >> Cross referenced git history and jira. Changelog needs some update
> >>
> >> Not in the release
> >>
> >>1. HDFS-16858 
> >>
> >>
> >>1. HADOOP-18532 
> >>2.
> >>   1. HDFS-16861 
> >>  2.
> >> 1. HDFS-16866
> >> 
> >> 2.
> >>1. HADOOP-18320
> >>
> >>2.
> >>
> >> Updated fixed version. Will generate. new Changelog in the next RC.
> >>
> >> Was able to build HBase and hbase-filesystem without any code change.
> >>
> >> hbase has one unit test failure. This one is reproducible even with
> >> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> >>
> >> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> >> 9.007 s <<< FAILURE! - in
> >> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> >> [ERROR]
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> >>  Time elapsed: 3.13 s  <<< ERROR!
> >> java.lang.OutOfMemoryError: Java heap space
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> >>
> >> hbase-filesystem has three test failures in TestHBOSSContractDistCp, and
> >> is not reproducible with Hadoop 3.3.5.
> >> [ERROR] Failures: [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:976->Assert.fail:88
> >> 10 errors in file of length 10
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureNoChange:270->AbstractContractDistCpTest.assertCounterInRange:290->Assert.assertTrue:41->Assert.fail:88
> >> Files Skipped value 0 too below minimum 1
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureToRemote:259->AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure:334->AbstractContractDistCpTest.assertCounterInRange:294->Assert.assertTrue:41->Assert.fail:88
> >> Files Copied value 2 above maximum 1
> >> [INFO]
> >> [ERROR] Tests run: 240, Failures: 3, Errors: 0, Skipped: 58
> >>
> >>
> >> Ozone
> >> test in progress. Will report back.
> >>
> >>
> >> On Tue, Jun 13, 2023 at 11:27 PM Wei-Chiu Chuang 
> >> wrote:
> >>
> >>> I am inviting anyone to try and vote on this release candidate.
> >>>
> >>> Note:
> >>> This is built off branch-3.3.6 plus PR#5741 (aws sdk update) and
> PR#5740
> >>> (LICENSE file update)
> >>>
> >>> The RC is available at:
> >>> 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Steve Loughran]

Which branch is -3.3.6 off? 3.3.5 or 3.3?

I'm travelling for the next few days and unlikely to be able to test this;
will do my best

On Wed, 14 Jun 2023 at 07:27, Wei-Chiu Chuang  wrote:

> I am inviting anyone to try and vote on this release candidate.
>
> Note:
> This is built off branch-3.3.6 plus PR#5741 (aws sdk update) and PR#5740
> (LICENSE file update)
>
> The RC is available at:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-amd64/ (for amd64)
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-arm64/ (for arm64)
>
> Git tag: release-3.3.6-RC0
> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC0
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1378/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC0-amd64/RELEASENOTES.md
>
> This is a relatively small release (by Hadoop standard) containing about
> 120 commits.
> Please give it a try, this RC vote will run for 7 days.
>
>
> Feature highlights:
>
> SBOM artifacts
> 
> Starting from this release, Hadoop publishes Software Bill of Materials
> (SBOM) using
> CycloneDX Maven plugin. For more information about SBOM, please go to
> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
>
> HDFS RBF: RDBMS based token storage support
> 
> HDFS Router-Router Based Federation now supports storing delegation tokens
> on MySQL,
> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> which improves token operation through over the original Zookeeper-based
> implementation.
>
>
> New File System APIs
> 
> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
> number of
> HDFS-specific APIs to Hadoop Common to make it possible for certain
> applications that
> depend on HDFS semantics to run on other Hadoop compatible file systems.
>
> In particular, recoverLease() and isFileClosed() are exposed through
> LeaseRecoverable
> interface. While setSafeMode() is exposed through SafeMode interface.
>

Re: [DISCUSS] Hadoop 3.3.6 release planning

[Steve Loughran]

Wei-Chiu has suggested a minimal "things in 3.3.5 which were very broken,
api change for ozone and any critical jar updates"

so much lower risk/easier to qualify and ship.

I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in here;
maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
remove json-smart...)

I'd also like to downgrade protobuf 2.5 from required to optional; even
though hadoop uses the shaded one, to support hbase etc the IPC code still
has direct use of the 2.5 classes. that coud be optional

if anyone wants to take up this PR, I would be very happy
https://github.com/apache/hadoop/pull/4996

On Fri, 5 May 2023 at 04:27, Xiaoqiao He  wrote:

> Thanks Wei-Chiu for driving this release.
> Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed version.
>
> so far only 8 jiras were resolved in the branch-3.3 line.
>
>
> If we should consider both 3.3.6 and 3.3.9 (which is from release-3.3.5
> discuss)[1] for this release line?
> I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> fixVersion in (3.3.6, 3.3.9)`[2],
> there are more than hundred jiras now.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> [2]
>
> https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
>
>
> On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang  wrote:
>
> > Hi community,
> >
> > I'd like to kick off the discussion around Hadoop 3.3.6 release plan.
> >
> > I'm being selfish but my intent for 3.3.6 is to have the new APIs in
> > HADOOP-18671  added
> so
> > we can have HBase to adopt this new API. Other than that, perhaps
> > thirdparty dependency updates.
> >
> > If you have open items to be added in the coming weeks, please add 3.3.6
> to
> > the target release version. Right now I am only seeing three open jiras
> > targeting 3.3.6.
> >
> > I imagine this is going to be a small release as 3.3.5 (hat tip to Steve)
> > was only made two months back, and so far only 8 jiras were resolved in
> the
> > branch-3.3 line.
> >
> > Best,
> > Weichiu
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

go ahead and cut it...I'd forgotten about it.

thanks,

steve

On Fri, 31 Mar 2023 at 06:44, Ayush Saxena  wrote:

> We have a daily build running for 3.3.5:
> https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/
>
> We have already released it, so I feel we can disable it. Will do it
> tomorrow, if nobody objects. In case the one who configured it wants
> to do it early, feel free to do so.
>
> We already have one for branch-3.3 which runs weekly which most
> probably most of us don't follow :)
>
> -Ayush
>
> On Wed, 22 Mar 2023 at 00:20, Steve Loughran
>  wrote:
> >
> > ok, here's my summary, even though most of the binding voters forgot to
> > declare they were on the PMC.
> >
> > +1 binding
> >
> > Steve Loughran
> > Chris Nauroth
> > Masatake Iwasaki
> > Ayush Saxena
> > Xiaoqiao He
> >
> > +1 non-binding
> >
> > Viraj Jasani
> >
> >
> > 0 or -1 votes: none.
> >
> >
> > Accordingly: the release is good!
> >
> > I will send the formal announcement out tomorrow
> >
> > A big thank you to everyone who qualified the RC, I know its a lot of
> work.
> > We can now get this out and *someone else* can plan the followup.
> >
> >
> > steve
> >
> > On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:
> >
> > > +1
> > >
> > > Thank you for the release candidate, Steve!
> > >
> > > * Verified all checksums.
> > > * Verified all signatures.
> > > * Built from source, including native code on Linux.
> > > * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> > > -Drequire.zstd -DskipTests
> > > * Tests passed.
> > > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > * Checked dependency tree to make sure we have all of the expected
> library
> > > updates that are mentioned in the release notes.
> > > * mvn -o dependency:tree
> > > * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> > > with no code.
> > > * For ARM verification:
> > > * Ran "file " on all native binaries in the ARM tarball to
> confirm
> > > they actually came out with ARM as the architecture.
> > > * Output of hadoop checknative -a on ARM looks good.
> > > * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> > > it worked fine.
> > > * Ran a MapReduce job with YARN configured to use
> > > LinuxContainerExecutor and verified launching the containers through
> > > container-executor worked.
> > >
> > > Chris Nauroth
> > >
> > >
> > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena 
> wrote:
> > >
> > > > +1(Binding)
> > > >
> > > > * Built from source (x86 & ARM)
> > > > * Successful Native Build (x86 & ARM)
> > > > * Verified Checksums (x86 & ARM)
> > > > * Verified Signature (x86 & ARM)
> > > > * Checked the output of hadoop version (x86 & ARM)
> > > > * Verified the output of hadoop checknative (x86 & ARM)
> > > > * Ran some basic HDFS shell commands.
> > > > * Ran some basic Yarn shell commands.
> > > > * Played a bit with HDFS Erasure Coding.
> > > > * Ran TeraGen & TeraSort
> > > > * Browed through NN, DN, RM & NM UI
> > > > * Skimmed over the contents of website.
> > > > * Skimmed over the contents of maven repo.
> > > > * Selectively ran some HDFS & CloudStore tests
> > > >
> > > > Thanx Steve for driving the release. Good Luck!!!
> > > >
> > > > -Ayush
> > > >
> > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> > > wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > * Verified signature and checksum of the source tarball.
> > > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean
> package
> > > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > > * Setup pseudo cluster with HDFS and YARN.
> > > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check
> the
> > > > > result.
> > > > > * Run example mr applications and check the result - Pi &
> wordcount.
> 

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

[Steve Loughran]

I do have a WiP library to hide that hoop-jumping behind a normal API, with
a goal of 3.2+ support only. It does actually compile against hadoop 2, but
it isn't tested there

https://github.com/steveloughran/fs-api-shim

1. my goal is to make it a hadoop library like hadoop-third-party, with its
own release cycle etc
2. testing is complex as it needs to have contract tests (lifted from
hadoop-common) applied through all versions of hadoop

so far openFile(), ByteBufferPositionedReadable, PathCapabilities *and*
VectorIO are in, though the latter has no tests yet.

If we can get this to work then (a) HBase can delegate suffering and (b)
libraries like parquet & ORC can use vector IO while still compiling
against older versions

On Tue, 28 Mar 2023 at 19:02, Nick Dimiduk  wrote:

> On Mon, Mar 27, 2023 at 20:29 Wei-Chiu Chuang  wrote:
>
> > For complex applications such as
> > HBase it is almost impossible to achieve true FS agnosticity without
> proper
> > contract tests, as now I am starting to realize.
> >
>
> This is absolutely true. HBase jumps through all sorts of painful
> reflective hoops to achieve reliable behavior across Hadoop versions.
> Steve’s proposal of self-describing APIs over opaque implementations would
> be drastically better than our current approach of reflection for
> inspecting and interacting with the internal details of any given client
> implementation.
>
> Thank you very much for your studious pursuit of this goal.
>
> Thanks,
> Nick
>
> On Mon, Mar 27, 2023 at 4:58 AM Steve Loughran  >
> > wrote:
> >
> > > side issue, as i think about what bulk delete call would also keep
> hbase
> > > happy
> > > https://issues.apache.org/jira/browse/HADOOP-18679
> > >
> > > should we think about new API calls only raising RuntimeExceptions?
> > >
> > > The more work I do on futures the more the way we always raise IOEs
> > > complicates life. java has outgrown checked exceptions
> > >
> > > On Fri, 24 Mar 2023 at 09:44, Steve Loughran 
> > wrote:
> > >
> > > >
> > > >
> > > > On Thu, 23 Mar 2023 at 10:07, Ayush Saxena 
> wrote:
> > > >
> > > >>
> > > >> Second idea mentioned in the original mail is also similar to
> > mentioned
> > > in
> > > >> the comment in the above ticket and is still quite acceptable, name
> > can
> > > be
> > > >> negotiated though, Add an interface to pull the relevant methods up
> in
> > > >> that
> > > >> without touching FileSystem class, we can have DFS implement that
> and
> > > >> Ozone
> > > >> FS implement them as well. We should be sorted: No Hacking, No
> > Bothering
> > > >> FileSystem and still things can work
> > > >>
> > > >>
> > > >>
> > > > This is the way we should be thinking about it. an interface which
> > > > filesystems MAY implement, but many do not.
> > > >
> > > > this has happened with some of the recent apis.
> > > >
> > > > presence of the API doesn't guarantee the api is active, only that it
> > may
> > > > be possible to call...callers should use PathCapabilities api to see
> if
> > > it
> > > > is live
> > > >
> > > >
> > > >>
> > >
> >
>

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

[Steve Loughran]

well, how about we flip the switch and get on with it.

slf4j seems happy on java11,

side issue, anyone seen test failures on zulu1.8; somehow my test run is
failing and i'm trying to work out whether its a mismatch in command
line/ide jvm versions, or the 3.3.5 JARs have been built with an openjdk
version which requires IntBuffer implements an overridden method IntBuffer
rewind().

java.lang.NoSuchMethodError: java.nio.IntBuffer.rewind()Ljava/nio/IntBuffer;

at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:341)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:308)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:257)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:202)
at java.io.DataInputStream.read(DataInputStream.java:149)

On Tue, 28 Mar 2023 at 15:52, Viraj Jasani  wrote:

> IIRC some of the ongoing major dependency upgrades (log4j 1 to 2, jersey 1
> to 2 and junit 4 to 5) are blockers for java 11 compile + test stability.
>
>
> On Tue, Mar 28, 2023 at 4:55 AM Steve Loughran  >
> wrote:
>
> >  Now that hadoop 3.3.5 is out, i want to propose something new
> >
> > we switch branch-3.3 and trunk to being java11 only
> >
> >
> >1. java 11 has been out for years
> >2. oracle java 8 is no longer available under "premier support"; you
> >can't really get upgrades
> >https://www.oracle.com/java/technologies/java-se-support-roadmap.html
> >3. openJDK 8 releases != oracle ones, and things you compile with them
> >don't always link to oracle java 8 (some classes in java.nio have
> added
> >more overrides)
> >4. more and more libraries we want to upgrade to/bundle are java 11
> only
> >5. moving to java 11 would cut our yetus build workload in half, and
> >line up for adding java 17 builds instead.
> >
> >
> > I know there are some outstanding issues still in
> > https://issues.apache.org/jira/browse/HADOOP-16795 -but are they
> blockers?
> > Could we just move to java11 and enhance at our leisure, once java8 is no
> > longer a concern.
> >
>

[DISCUSS] hadoop branch-3.3+ going to java11 only

[Steve Loughran]

 Now that hadoop 3.3.5 is out, i want to propose something new

we switch branch-3.3 and trunk to being java11 only


   1. java 11 has been out for years
   2. oracle java 8 is no longer available under "premier support"; you
   can't really get upgrades
   https://www.oracle.com/java/technologies/java-se-support-roadmap.html
   3. openJDK 8 releases != oracle ones, and things you compile with them
   don't always link to oracle java 8 (some classes in java.nio have added
   more overrides)
   4. more and more libraries we want to upgrade to/bundle are java 11 only
   5. moving to java 11 would cut our yetus build workload in half, and
   line up for adding java 17 builds instead.


I know there are some outstanding issues still in
https://issues.apache.org/jira/browse/HADOOP-16795 -but are they blockers?
Could we just move to java11 and enhance at our leisure, once java8 is no
longer a concern.

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

[Steve Loughran]

side issue, as i think about what bulk delete call would also keep hbase
happy
https://issues.apache.org/jira/browse/HADOOP-18679

should we think about new API calls only raising RuntimeExceptions?

The more work I do on futures the more the way we always raise IOEs
complicates life. java has outgrown checked exceptions

On Fri, 24 Mar 2023 at 09:44, Steve Loughran  wrote:

>
>
> On Thu, 23 Mar 2023 at 10:07, Ayush Saxena  wrote:
>
>>
>> Second idea mentioned in the original mail is also similar to mentioned in
>> the comment in the above ticket and is still quite acceptable, name can be
>> negotiated though, Add an interface to pull the relevant methods up in
>> that
>> without touching FileSystem class, we can have DFS implement that and
>> Ozone
>> FS implement them as well. We should be sorted: No Hacking, No Bothering
>> FileSystem and still things can work
>>
>>
>>
> This is the way we should be thinking about it. an interface which
> filesystems MAY implement, but many do not.
>
> this has happened with some of the recent apis.
>
> presence of the API doesn't guarantee the api is active, only that it may
> be possible to call...callers should use PathCapabilities api to see if it
> is live
>
>
>>

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

[Steve Loughran]

On Thu, 23 Mar 2023 at 10:07, Ayush Saxena  wrote:

>
> Second idea mentioned in the original mail is also similar to mentioned in
> the comment in the above ticket and is still quite acceptable, name can be
> negotiated though, Add an interface to pull the relevant methods up in that
> without touching FileSystem class, we can have DFS implement that and Ozone
> FS implement them as well. We should be sorted: No Hacking, No Bothering
> FileSystem and still things can work
>
>
>
This is the way we should be thinking about it. an interface which
filesystems MAY implement, but many do not.

this has happened with some of the recent apis.

presence of the API doesn't guarantee the api is active, only that it may
be possible to call...callers should use PathCapabilities api to see if it
is live


>

[ANNOUNCE] Apache Hadoop 3.3.5 release

[Steve Loughran]

On behalf of the Apache Hadoop Project Management Committee, I am
pleased to announce the release of Apache Hadoop 3.3.5.


This is a release of Apache Hadoop 3.3 line.

Key changes include

* A big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native IO
  s3a:// parallel GET requests.
* Arm64 binaries. Note, because the arm64 release was on a different
  platform, the jar files may not match those of the x86
  release -and therefore the maven artifacts.
* Security fixes in Hadoop's own code.

Users of Apache Hadoop 3.3.4 and earlier should upgrade to
this release.

All users are encouraged to read the [overview of major changes][1]
since release 3.3.4.

For details of bug fixes, improvements, and other enhancements since
the previous 3.3.4 release, please check [release notes][2]
and [changelog][3].


*Azure ABFS: Critical Stream Prefetch Fix*


The ABFS connector has a critical bug fix
https://issues.apache.org/jira/browse/HADOOP-18546:
*ABFS. Disable purging list of in-progress reads in abfs stream close().*

All users of the abfs connector in hadoop releases 3.3.2+ MUST either
upgrade
to this release or disable prefetching by setting
`fs.azure.readaheadqueue.depth` to `0`.


[1]: http://hadoop.apache.org/docs/r3.3.5/index.html
[2]:
http://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-common/release/3.3.5/RELEASENOTES.3.3.5.html
[3]:
http://hadoop.apache.org/docs/r3.3.5/hadoop-project-dist/hadoop-common/release/3.3.5/CHANGELOG.3.3.5.html


Many thanks to everyone who helped in this release by supplying patches,
reviewing them, helping get this release building and testing and
reviewing the final artifacts.

steve

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

ok, here's my summary, even though most of the binding voters forgot to
declare they were on the PMC.

+1 binding

Steve Loughran
Chris Nauroth
Masatake Iwasaki
Ayush Saxena
Xiaoqiao He

+1 non-binding

Viraj Jasani


0 or -1 votes: none.


Accordingly: the release is good!

I will send the formal announcement out tomorrow

A big thank you to everyone who qualified the RC, I know its a lot of work.
We can now get this out and *someone else* can plan the followup.


steve

On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:

> +1
>
> Thank you for the release candidate, Steve!
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file " on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> Chris Nauroth
>
>
> On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena  wrote:
>
> > +1(Binding)
> >
> > * Built from source (x86 & ARM)
> > * Successful Native Build (x86 & ARM)
> > * Verified Checksums (x86 & ARM)
> > * Verified Signature (x86 & ARM)
> > * Checked the output of hadoop version (x86 & ARM)
> > * Verified the output of hadoop checknative (x86 & ARM)
> > * Ran some basic HDFS shell commands.
> > * Ran some basic Yarn shell commands.
> > * Played a bit with HDFS Erasure Coding.
> > * Ran TeraGen & TeraSort
> > * Browed through NN, DN, RM & NM UI
> > * Skimmed over the contents of website.
> > * Skimmed over the contents of maven repo.
> > * Selectively ran some HDFS & CloudStore tests
> >
> > Thanx Steve for driving the release. Good Luck!!!
> >
> > -Ayush
> >
> > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> wrote:
> > >
> > > +1
> > >
> > > * Verified signature and checksum of the source tarball.
> > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > > -DskipTests -Pnative -Pdist -Dtar`.
> > > * Setup pseudo cluster with HDFS and YARN.
> > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> > > result.
> > > * Run example mr applications and check the result - Pi & wordcount.
> > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> > >
> > > Thanks Steve for your work.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> > iwasak...@oss.nttdata.com>
> > >> wrote:
> > >>
> > >> +1
> > >>
> > >> + verified the signature and checksum of the source tarball.
> > >>
> > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK
> 8
> > >> with native profile enabled.
> > >>   + launched pseudo distributed cluster including kms and httpfs with
> > >> Kerberos and SSL enabled.
> > >>   + created encryption zone, put and read files via httpfs.
> > >>   + ran example MR wordcount over encryption zone.
> > >>   + checked the binary of container-executor.
> > >>
> > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky
> > Linux
> > >> 8 (aarch64).
> > >>   + ran smoke-tests of hdfs, yarn and mapreduce.
> > >> + built site documentation and skimmed the contents.
> > >>   +  Javadocs are contained.
> > >>
> > >> Thanks,
> > >> Masatake Iwasaki
> > >>
> > >>> On 2023/03/16 4:47, Steve Loughran wrote:
> > >>> Apache Hadoop 3.3.5
> > >>>
> > >>> Mukund and I have put toget

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Thank you for this!

Can anyone else with time do a review too? i really want to get this one
done, now the HDFS issues are all resolved.

I do not want this release to fall by the wayside through lack of votes
alone. In fact, I would be very unhappy



On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:

> +1 (non-binding)
>
> * Signature/Checksum: ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> Containerized deployments:
> * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
> * Deployed and started JHS, RM, NM
> * Hbase, hdfs CRUD looks good
> * Sample RowCount MapReduce job looks good
>
> * S3A tests with scale profile looks good
>
>
> On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> 
> wrote:
>
> > Apache Hadoop 3.3.5
> >
> > Mukund and I have put together a release candidate (RC3) for Hadoop
> 3.3.5.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> >
> > The git tag is release-3.3.5-RC3, commit 706d88266ab
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >   transitive CVEs under control -both genuine and false positives.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >   high-performance versions for file:// and s3a:// filesystems.
> >   file:// through java native io
> >   s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >   compatible systems validate these.
> > * and compared to the previous RC, all the major changes are
> >   HDFS issues.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

and my vote

My vote

+1 binding

I've been using the RCs for a while as my CLI entry point, and testing it
through other builds

for this RC
* Local builds of cloudstore
* fs-api-shim
* spark
* built and ran my cloud integration tests, which now include large CVS
file jobs which should show the Azure prefetch bug if it still existed.

downloaded the tar, expanded it, ran command line code with it, including
cloudstore against the stores. we need to get hadoop-azure and its
dependencies onto the path by default, to make abfs io easier.


I have the arm binaries building, and did a checknative to make sure all
was good

stevel@0da162643f99:~/hadoop/patchprocess/hadoop-3.3.5$ bin/hadoop
checknative
2023-03-17 13:00:27,107 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2023-03-17 13:00:27,112 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2023-03-17 13:00:27,121 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2023-03-17 13:00:27,156 INFO nativeio.NativeIO: The native code was built
without PMDK support.
Native library checking:
hadoop:  true
/home/stevel/hadoop/patchprocess/hadoop-3.3.5/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: true /lib/aarch64-linux-gnu/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

---


On Wed, 15 Mar 2023 at 19:47, Steve Loughran  wrote:

>
> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>
> The git tag is release-3.3.5-RC3, commit 706d88266ab
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
> * and compared to the previous RC, all the major changes are
>   HDFS issues.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

[Steve Loughran]

   1. I think a new interface would be good as FileContext could do the
   same thing
   2. using PathCapabilities probes should still be mandatory as for
   FileContext it would depend on the back end
   3. Whoever does this gets to specify what the API does and write the
   contract tests. Saying "just to do what HDFS does" isn't enough as it's not
   always clear the HDFS team no how much of that behaviour is intentional
   (rename, anyone?).


For any new API (a better rename, a better delete,...) I would normally
insist on making it cloud friendly, with an extensible builder API and an
emphasis on asynchronous IO. However this is existing code and does target
HDFS and Ozone -pulling the existing APIs up into a new interface seems the
right thing to do here.

 I have a WiP project to do a shim library to offer new FS APIs two older
Hadoop releases by way of reflection, so that we can get new APIs taken up
across projects where we cannot choreograph version updates across the
entire stack. (hello parquet, spark,...). My goal is to actually make this
a Hadoop managed project, with its own release schedule. You could add an
equivalent of the new interface in here, which would then use reflection
behind-the-scenes to invoke the underlying HDFS methods when the FS client
has them.

https://github.com/steveloughran/fs-api-shim

I've just added vector IO API there; the next step is to copy over a lot of
the contract tests from hadoop common and apply them through the shim -to
hadoop 3.2, 3.3.0-3.3.5. That testing against many backends is actually as
tricky as the reflection itself. However without this library it is going
to take a long long time for the open source applications to pick up the
higher performance/Cloud ready Apis. Yes, those of us who can build the
entire stack can do it, but that gradually adds more divergence from the
open source libraries, reduces the test coverage overall and only increases
maintenance costs over time.

steve

On Thu, 16 Mar 2023 at 20:56, Wei-Chiu Chuang  wrote:

> Hi,
>
> Stephen and I are working on a project to make HBase to run on Ozone.
>
> HBase, born out of the Hadoop project, depends on a number of HDFS specific
> APIs, including recoverLease() and isInSafeMode(). The HBase community [1]
> strongly voiced that they don't want the project to have direct dependency
> on additional FS implementations due to dependency and vulnerability
> management concerns.
>
> To make this project successful, we're exploring options, to push up these
> APIs to the FileSystem abstraction. Eventually, it would make HBase FS
> implementation agnostic, and perhaps enable HBase to support other storage
> systems in the future.
>
> We'd use the PathCapabilities API to probe if the underlying FS
> implementation supports these APIs, and would then invoke the corresponding
> FileSystem APIs. This is straightforward but the FileSystem would become
> bloated.
>
> Another option is to create a "RecoverableFileSystem" interface, and have
> both DistributedFileSystem (HDFS) and RootedOzoneFileSystem (Ozone). This
> way the impact to the Hadoop project and the FileSystem abstraction is even
> smaller.
>
> Thoughts?
>
> [1] https://lists.apache.org/thread/tcrp8vxxs3z12y36mpzx35txhpp7tvxv
>

[VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/

The git tag is release-3.3.5-RC3, commit 706d88266ab

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.
* and compared to the previous RC, all the major changes are
  HDFS issues.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

thanks.

now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week

i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months

On Mon, 6 Mar 2023 at 23:29, Erik Krogen  wrote:

> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1...@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zande...@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu ]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> ste...@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran  >
> wrote:
>
> >  i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > 
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena 
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HD

[jira] [Resolved] (HDFS-16934) org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16934.
---
Fix Version/s: 3.4.0
   3.3.5
   Resolution: Fixed

fixed; ran new test on 3.3.5 to verify the backport/conflict resolution was ok.

> org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
> -
>
> Key: HDFS-16934
> URL: https://issues.apache.org/jira/browse/HDFS-16934
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: dfsadmin, test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>    Reporter: Steve Loughran
>Assignee: Shilun Fan
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> jenkins test failure as the logged output is in the wrong order for the 
> assertions. HDFS-16624 flipped the order...without that this would have 
> worked.
> {code}
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:87)
>   at org.junit.Assert.assertTrue(Assert.java:42)
>   at org.junit.Assert.assertTrue(Assert.java:53)
>   at 
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1149)
> {code}
> Here the code is asserting about the contents of the output, 
> {code}
> assertTrue(outs.get(0).startsWith("Reconfiguring status for node"));
> assertTrue("SUCCESS: Changed property 
> dfs.datanode.peer.stats.enabled".equals(outs.get(2))
> || "SUCCESS: Changed property 
> dfs.datanode.peer.stats.enabled".equals(outs.get(1)));  // here
> assertTrue("\tFrom: \"false\"".equals(outs.get(3)) || "\tFrom: 
> \"false\"".equals(outs.get(2)));
> assertTrue("\tTo: \"true\"".equals(outs.get(4)) || "\tTo: 
> \"true\"".equals(outs.get(3)))
> {code}
> If you look at the log, the actual line is appearing in that list, just in a 
> different place. race condition
> {code}
> 2023-02-24 01:02:06,275 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin (TestDFSAdmin.java:testAllDatanodesReconfig(1146)) - 
> dfsadmin -status -livenodes output:
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring 
> status for node [127.0.0.1:41795]: started at Fri Feb 24 01:02:03 GMT 2023 
> and finished at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring 
> status for node [127.0.0.1:34007]: started at Fri Feb 24 01:02:03 GMT 
> 2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -  From: "false"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -  To: "true"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -  and finished 
> at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  
> tools.TestDFSAdmin 
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - SUCCESS: 
> Changed property dfs.datanode.peer.stats.enabled
> {code}
> we have a race condition in output generation and the assertions are clearly 
> too brittle
> for the 3.3.5 release I'm not going to make this a blocker. What i will do is 
> propose that the asserts move to assertJ with an assertion that the 
> collection "containsExactlyInAnyOrder" all the strings.
> That will
> 1. not be brittle.
> 2. give nice errors on failure



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

 i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one

On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:

> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran  >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > 
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani 
> wrote:
> > >> &g

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:

> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>

thanks.

>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>

OK. Could you have a go with a (locally built) patch release

>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran 
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
>>
>> > I will highlight that I am completely fed up with doing this  release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >>  - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >>  - mvn clean install  -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >>  - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> 
>> >> wrote:
>> >>
>> >> 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.

On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:

> I will highlight that I am completely fed up with doing this  release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>>  - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>>  - mvn clean install  -DskipTests
>> * Built tar from source (1.8.0_341): ok
>>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> 
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> >issues in maven 3.9.0)
>> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

well, lets see what others say.

we don't want to ship stuff with serious regression to hdfs.

I will highlight that I am completely fed up with doing this  release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.

Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.

I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.

Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.



On Thu, 2 Mar 2023 at 00:27, Erik Krogen  wrote:

> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> >issues in maven 3.9.0)
> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>

[jira] [Resolved] (HDFS-16935) TestFsDatasetImpl.testReportBadBlocks brittle

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16935.
---
Resolution: Fixed

> TestFsDatasetImpl.testReportBadBlocks brittle
> -
>
> Key: HDFS-16935
> URL: https://issues.apache.org/jira/browse/HDFS-16935
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>    Reporter: Steve Loughran
>Assignee: Viraj Jasani
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> jenkins failure as sleep() time not long enough
> {code}
> Failing for the past 1 build (Since #4 )
> Took 7.4 sec.
> Error Message
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:89)
>   at org.junit.Assert.failNotEquals(Assert.java:835)
>   at org.junit.Assert.assertEquals(Assert.java:647)
>   at org.junit.Assert.assertEquals(Assert.java:633)
> {code}
> assert is after a 3s sleep waiting for reports coming in.
> {code}
>   dataNode.reportBadBlocks(block, dataNode.getFSDataset()
>   .getFsVolumeReferences().get(0));
>   Thread.sleep(3000);   // 3s 
> sleep
>   BlockManagerTestUtil.updateState(cluster.getNamesystem()
>   .getBlockManager());
>   // Verify the bad block has been reported to namenode
>   Assert.assertEquals(1, 
> cluster.getNamesystem().getCorruptReplicaBlocks());  // here
> {code}
> LambdaTestUtils.eventually() should be used around this assert, maybe with an 
> even shorter initial delay so on faster systems, test is faster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.

We need anyone who can to verify the source and binary artifacts,
including those JARs staged on maven, the site documentation and the arm64
tar file.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/

The git tag is release-3.3.5-RC2, commit 72f8c2a4888

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

As to what changed since the RC1 attempt last week


   1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
   2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
   3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
   issues in maven 3.9.0)
   4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)


Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

 need this pr in too, https://github.com/apache/hadoop/pull/5429

   1. cuts back on some transitive dependencies from hadoop-aliyun
   2. fixes LICENSE-bin to be correct

#2 is the blocker...and it looks like 3.2.x will also need fixup as well as
the later ones -hadoop binaries have shipped without that file being up to
date, but at least all the transitive stuff is correctly licensed. And i
think we need to change the PR template to mention transitive updates in
the license bit too

if this goes in, I will do the rebuild on monday UK time

On Thu, 23 Feb 2023 at 11:18, Steve Loughran  wrote:

>
> And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
> maven releases (3.9.0)
>
> on a new local build with maven updated on homebrew (which i needed for
> spark). so a code change too. That issue doesn't surface on our
> release dockers, but will hit other people. especially over time. Going to
> revert HADOOP-18590. Publish SBOM artifacts (#5281)
>
>
>
> On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:
>
>> ok, let me cancel, update those jiras and kick off again. that will save
>> anyone else having to do their homework
>>
>> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
>> wrote:
>>
>>> I'm now -1 as I found the wrong information on the top page (index.md).
>>>
>>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>>> Router Based Federation.
>>>
>>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>>> though it is actually not in branch-3.3. I corrected the fix version and
>>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>>> Unfortunately, it won't be fixed soon. I should have let you know at that
>>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>>> So I think we have to delete the description from the top page not to
>>> confuse Hadoop users.
>>>
>>> - Takanobu
>>>
>>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>>
>>> > Thanks for driving the release, Steve and Mukund.
>>> >
>>> > I found that there were some jiras with wrong fix versions.
>>> >
>>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>>> > - HDFS-16845
>>> > - HADOOP-18345
>>> >
>>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>>> > (and it is not in release-3.3.4) :
>>> > - HADOOP-17276
>>> > - HDFS-13293
>>> > - HDFS-15630
>>> > - HDFS-16266
>>> > - HADOOP-18003
>>> > - HDFS-16310
>>> > - HADOOP-18014
>>> >
>>> > I corrected all the wrong fix versions just now. I'm not sure we should
>>> > revote it since it only affects the changelog.
>>> >
>>> > - Takanobu
>>> >
>>> > 2023年2月21日(火) 22:43 Steve Loughran :
>>> >
>>> >> Apache Hadoop 3.3.5
>>> >>
>>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>>> 3.3.5.
>>> >>
>>> >> What we would like is for anyone who can to verify the tarballs,
>>> >> especially
>>> >> anyone who can try the arm64 binaries as we want to include them too.
>>> >>
>>> >> The RC is available at:
>>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>>> >>
>>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>>> >>
>>> >> The maven artifacts are staged at
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>>> >>
>>> >> You can find my public key at:
>>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >>
>>> >> Change log
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>>> >>
>>> >> Release notes
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>>> >>
>>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>>> >>
>>> >> Key changes include
>>> >>
>>> >> * Big update of dependencies to try and keep those reports of
>>> >>   transitive CVEs under control -both genuine and false positives.
>>>

[jira] [Created] (HDFS-16935) TestFsDatasetImpl.testReportBadBlocks brittle

[Steve Loughran (Jira)]

Steve Loughran created HDFS-16935:
-

 Summary: TestFsDatasetImpl.testReportBadBlocks brittle
 Key: HDFS-16935
 URL: https://issues.apache.org/jira/browse/HDFS-16935
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.4.0, 3.3.5, 3.3.9
Reporter: Steve Loughran


jenkins failure as sleep() time not long enough
{code}
Failing for the past 1 build (Since #4 )
Took 7.4 sec.
Error Message
expected:<1> but was:<0>
Stacktrace
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
{code}

assert is after a 3s sleep waiting for reports coming in.
{code}
  dataNode.reportBadBlocks(block, dataNode.getFSDataset()
  .getFsVolumeReferences().get(0));
  Thread.sleep(3000);   // 3s sleep
  BlockManagerTestUtil.updateState(cluster.getNamesystem()
  .getBlockManager());
  // Verify the bad block has been reported to namenode
  Assert.assertEquals(1, 
cluster.getNamesystem().getCorruptReplicaBlocks());  // here
{code}

LambdaTestUtils.eventually() should be used around this assert, maybe with an 
even shorter initial delay so on faster systems, test is faster.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16934) org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression

[Steve Loughran (Jira)]

Steve Loughran created HDFS-16934:
-

 Summary: 
org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
 Key: HDFS-16934
 URL: https://issues.apache.org/jira/browse/HDFS-16934
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: dfsadmin, test
Affects Versions: 3.4.0, 3.3.5, 3.3.9
Reporter: Steve Loughran


jenkins test failure as the logged output is in the wrong order for the 
assertions. HDFS-16624 flipped the order...without that this would have worked.

{code}

java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at 
org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1149)
{code}


Here the code is asserting about the contents of the output, 
{code}
assertTrue(outs.get(0).startsWith("Reconfiguring status for node"));
assertTrue("SUCCESS: Changed property 
dfs.datanode.peer.stats.enabled".equals(outs.get(2))
|| "SUCCESS: Changed property 
dfs.datanode.peer.stats.enabled".equals(outs.get(1)));  // here
assertTrue("\tFrom: \"false\"".equals(outs.get(3)) || "\tFrom: 
\"false\"".equals(outs.get(2)));
assertTrue("\tTo: \"true\"".equals(outs.get(4)) || "\tTo: 
\"true\"".equals(outs.get(3)))
{code}

If you look at the log, the actual line is appearing in that list, just in a 
different place. race condition
{code}
2023-02-24 01:02:06,275 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:testAllDatanodesReconfig(1146)) - dfsadmin -status 
-livenodes output:
2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring 
status for node [127.0.0.1:41795]: started at Fri Feb 24 01:02:03 GMT 2023 and 
finished at Fri Feb 24 01:02:03 GMT 2023.
2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring 
status for node [127.0.0.1:34007]: started at Fri Feb 24 01:02:03 GMT 
2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -From: "false"
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -To: "true"
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -  and finished at 
Fri Feb 24 01:02:03 GMT 2023.
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO  tools.TestDFSAdmin 
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - SUCCESS: Changed 
property dfs.datanode.peer.stats.enabled
{code}
we have a race condition in output generation and the assertions are clearly 
too brittle

for the 3.3.5 release I'm not going to make this a blocker. What i will do is 
propose that the asserts move to assertJ with an assertion that the collection 
"containsExactlyInAnyOrder" all the strings.

That will
1. not be brittle.
2. give nice errors on failure




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
maven releases (3.9.0)

on a new local build with maven updated on homebrew (which i needed for
spark). so a code change too. That issue doesn't surface on our
release dockers, but will hit other people. especially over time. Going to
revert HADOOP-18590. Publish SBOM artifacts (#5281)



On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:

> ok, let me cancel, update those jiras and kick off again. that will save
> anyone else having to do their homework
>
> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
> wrote:
>
>> I'm now -1 as I found the wrong information on the top page (index.md).
>>
>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>> Router Based Federation.
>>
>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>> though it is actually not in branch-3.3. I corrected the fix version and
>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>> Unfortunately, it won't be fixed soon. I should have let you know at that
>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>> So I think we have to delete the description from the top page not to
>> confuse Hadoop users.
>>
>> - Takanobu
>>
>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>
>> > Thanks for driving the release, Steve and Mukund.
>> >
>> > I found that there were some jiras with wrong fix versions.
>> >
>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>> > - HDFS-16845
>> > - HADOOP-18345
>> >
>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>> > (and it is not in release-3.3.4) :
>> > - HADOOP-17276
>> > - HDFS-13293
>> > - HDFS-15630
>> > - HDFS-16266
>> > - HADOOP-18003
>> > - HDFS-16310
>> > - HADOOP-18014
>> >
>> > I corrected all the wrong fix versions just now. I'm not sure we should
>> > revote it since it only affects the changelog.
>> >
>> > - Takanobu
>> >
>> > 2023年2月21日(火) 22:43 Steve Loughran :
>> >
>> >> Apache Hadoop 3.3.5
>> >>
>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>> 3.3.5.
>> >>
>> >> What we would like is for anyone who can to verify the tarballs,
>> >> especially
>> >> anyone who can try the arm64 binaries as we want to include them too.
>> >>
>> >> The RC is available at:
>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>> >>
>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>> >>
>> >> The maven artifacts are staged at
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>> >>
>> >> You can find my public key at:
>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >>
>> >> Change log
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>> >>
>> >> Release notes
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>> >>
>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>> >>
>> >> Key changes include
>> >>
>> >> * Big update of dependencies to try and keep those reports of
>> >>   transitive CVEs under control -both genuine and false positives.
>> >> * HDFS RBF enhancements
>> >> * Critical fix to ABFS input stream prefetching for correct reading.
>> >> * Vectored IO API for all FSDataInputStream implementations, with
>> >>   high-performance versions for file:// and s3a:// filesystems.
>> >>   file:// through java native io
>> >>   s3a:// parallel GET requests.
>> >> * This release includes Arm64 binaries. Please can anyone with
>> >>   compatible systems validate these.
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote. Even automating that would be risky.
>> >>
>> >> Please try the release and vote. The vote will run for 5 days.
>> >>
>> >> Steve and Mukund
>> >>
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

ok, let me cancel, update those jiras and kick off again. that will save
anyone else having to do their homework

On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma  wrote:

> I'm now -1 as I found the wrong information on the top page (index.md).
>
> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
> Router Based Federation.
>
> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
> though it is actually not in branch-3.3. I corrected the fix version and
> created HDFS-16889 to backport them to branch-3.3 about a month ago.
> Unfortunately, it won't be fixed soon. I should have let you know at that
> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
> So I think we have to delete the description from the top page not to
> confuse Hadoop users.
>
> - Takanobu
>
> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>
> > Thanks for driving the release, Steve and Mukund.
> >
> > I found that there were some jiras with wrong fix versions.
> >
> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> > - HDFS-16845
> > - HADOOP-18345
> >
> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
> > (and it is not in release-3.3.4) :
> > - HADOOP-17276
> > - HDFS-13293
> > - HDFS-15630
> > - HDFS-16266
> > - HADOOP-18003
> > - HDFS-16310
> > - HADOOP-18014
> >
> > I corrected all the wrong fix versions just now. I'm not sure we should
> > revote it since it only affects the changelog.
> >
> > - Takanobu
> >
> > 2023年2月21日(火) 22:43 Steve Loughran :
> >
> >> Apache Hadoop 3.3.5
> >>
> >> Mukund and I have put together a release candidate (RC1) for Hadoop
> 3.3.5.
> >>
> >> What we would like is for anyone who can to verify the tarballs,
> >> especially
> >> anyone who can try the arm64 binaries as we want to include them too.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
> >>
> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
> >>
> >> The maven artifacts are staged at
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
> >>
> >> Release notes
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >>   transitive CVEs under control -both genuine and false positives.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>   high-performance versions for file:// and s3a:// filesystems.
> >>   file:// through java native io
> >>   s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >>   compatible systems validate these.
> >>
> >> Note, because the arm64 binaries are built separately on a different
> >> platform and JVM, their jar files may not match those of the x86
> >> release -and therefore the maven artifacts. I don't think this is
> >> an issue (the ASF actually releases source tarballs, the binaries are
> >> there for help only, though with the maven repo that's a bit blurred).
> >>
> >> The only way to be consistent would actually untar the x86.tar.gz,
> >> overwrite its binaries with the arm stuff, retar, sign and push out
> >> for the vote. Even automating that would be risky.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> Steve and Mukund
> >>
> >
>

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/

The git tag is release-3.3.5-RC1, commit 274f91a3259

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1368/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

[jira] [Resolved] (HDFS-16853) The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16853.
---
Fix Version/s: 3.3.5
   Resolution: Fixed

> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed 
> because HADOOP-18324
> ---
>
> Key: HDFS-16853
> URL: https://issues.apache.org/jira/browse/HDFS-16853
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.3.5
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 3.3.5
>
>
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed 
> with error message: Waiting for cluster to become active. And the blocking 
> jstack as bellows:
> {code:java}
> "BP-1618793397-192.168.3.4-1669198559828 heartbeating to 
> localhost/127.0.0.1:54673" #260 daemon prio=5 os_prio=31 tid=0x
> 7fc1108fa000 nid=0x19303 waiting on condition [0x700017884000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x0007430a9ec0> (a 
> java.util.concurrent.SynchronousQueue$TransferQueue)
>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>         at 
> java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(SynchronousQueue.java:762)
>         at 
> java.util.concurrent.SynchronousQueue$TransferQueue.transfer(SynchronousQueue.java:695)
>         at 
> java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:877)
>         at 
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1186)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1482)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1429)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
>         at 
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
>         at com.sun.proxy.$Proxy23.sendHeartbeat(Unknown Source)
>         at 
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClient
> SideTranslatorPB.java:168)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:570)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:714)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:915)
>         at java.lang.Thread.run(Thread.java:748)  {code}
> After looking into the code and found that this bug is imported by 
> HADOOP-18324. Because RpcRequestSender exited without cleaning up the 
> rpcRequestQueue, then caused BPServiceActor was blocked in sending request.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

thanks

pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.

i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts

On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki 
wrote:

> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module

hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework


>[INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
>[ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>Please check the following and either correct the build or update
>the allowed list with reasoning.
>
>com/
>com/sun/
>com/sun/jersey/
>com/sun/jersey/json/
>...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc  might be the fix.
>
>[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
>...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563  ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR

>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> >+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> >+ created encryption zone, put and read files via httpfs.
> >+ ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >  # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> >- Javadocs are not contained (under r3.3.5/api).
> >  # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and
expect people to be able to test it thoroughly enough to make this the one
we can ship.

What we would like is for anyone who can to verify the tarballs, and test
the binaries, especially anyone who can try the arm64 binaries. We've got
the building of those done and now the build file will incorporate them
into the release -but neither of us have actually tested it yet. Maybe I
should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones
from x86 build. Building and testing downstream apps will be incredibly
helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495904d

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.


Please try the release and vote on it, even though i don't know what is a
good timeline here...i'm actually going on holiday in early jan. Mukund is
around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan
for that reason

Steve (and mukund)

exciting new content needed for the 3.3.5 index.md file

[Steve Loughran]

The first "smoke test" RC is about to be up for people to play with, we are
just testing things here and getting that arm build done.

Can I have some content for the index.html page describing what has changed?
hadoop-project/src/site/markdown/index.md.vm

I can (and will) speak highly of stuff I've been involved in, but need
contributions from others for what is new in this release in HDFS, YARN,
and MR (other than the manifest committer).

It'd be good to have a list of CVEs fixed by upgrading jars. Maybe we
should have a transitive-CVE tag for all JIRAs which update a dependency
for this, so that then we could have the release notes explicitly list
these in their own section.

Please submit changes to branch-3.3.5; use HADOOP-18470. as the jira for
all the release notes.

 thanks.

[jira] [Resolved] (HDFS-16795) Use secure XML parser utils in hdfs classes

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16795.
---
Fix Version/s: 3.3.5
   Resolution: Fixed

> Use secure XML parser utils in hdfs classes
> ---
>
> Key: HDFS-16795
> URL: https://issues.apache.org/jira/browse/HDFS-16795
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: PJ Fanning
>Assignee: PJ Fanning
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.3.9
>
>
> Uptakes HADOOP-18469



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.5 release planning

[Steve Loughran]

On Fri, 7 Oct 2022 at 22:36, Wei-Chiu Chuang  wrote:

> Bumping this up. Adding the [DISCUSS] text to make this message stand out
> of your inbox.
>
> I certainly missed this message and only realized 3.3.5 has more than just
> security updates.
>
> What was the issue with the ARM64 build? I was able to publish ARM64 build
> for 3.3.1 release without problems.
>

changes in the underlying dependencies, python libraries in particular.
things wouldn't build on branch-3.3.

Also, building on a macbook m1 wasn't a problem then -it is now and some of
the bash tests needed tuning

fixed now HADOOP-18401



>
>
> On Tue, Sep 27, 2022 at 9:35 AM Steve Loughran  >
> wrote:
>
> > Mukund has just created the new Hadoop release JIRA,
> > https://issues.apache.org/jira/browse/HADOOP-18470, and is doing the
> first
> > build/test before pushing it up. This is off branch-3.3, so it will have
> > more significant changes than the 3.3.3 and 3.3.4 releases, which were
> just
> > CVE/integration fixes.
> >
> > The new branch, branch-3.3.5 has its maven/hadoop.version set to
> > 3.3.5-SNAPSHOT.
> >
> > All JIRA issues fixed/blocked for 3.3.9 now reference 3.3.5. The next
> step
> > of the release is to actually move those wontfix issues back to being
> > against 3.3.9
> >
> > There is still a 3.3.9 version; branch-3.3's maven build still refers to
> > it. Issues found/fixed in branch-3.3 *but not the new branch-3.3.5
> branch-
> > should still refer to this version. Anything targeting the 3.3.5 release
> > must be committed to the new branch, and have its JIRA version tagged
> > appropriately.
> >
> > All changes to be cherrypicked into 3.3.5, except for those ones related
> to
> > the release process itself, MUST be in branch-3.3 first, and SHOULD be in
> > trunk unless there is some fundamental reason they can't apply there
> > (reload4j etc).
> >
> > Let's try and stabilise this releases, especially bringing up to date all
> > the JAR dependencies which we can safely update.
> >
> > Anyone planning to give talks at ApacheCon about forthcoming features
> > already in 3.3 SHOULD
> >
> >1. reference Hadoop 3.3.5 as the version
> >2. make sure their stuff works.
> >
> > Mukund will be at the conf; find him and offer any help you can in
> getting
> > this release out.
> >
> > I'd like to get that Arm64 build workingdoes anyone else want to get
> > involved?
> >
> > -steve
> >
>

HADOOP-18470 Release hadoop 3.3.5

[Steve Loughran]

Mukund has just created the new Hadoop release JIRA,
https://issues.apache.org/jira/browse/HADOOP-18470, and is doing the first
build/test before pushing it up. This is off branch-3.3, so it will have
more significant changes than the 3.3.3 and 3.3.4 releases, which were just
CVE/integration fixes.

The new branch, branch-3.3.5 has its maven/hadoop.version set to
3.3.5-SNAPSHOT.

All JIRA issues fixed/blocked for 3.3.9 now reference 3.3.5. The next step
of the release is to actually move those wontfix issues back to being
against 3.3.9

There is still a 3.3.9 version; branch-3.3's maven build still refers to
it. Issues found/fixed in branch-3.3 *but not the new branch-3.3.5 branch-
should still refer to this version. Anything targeting the 3.3.5 release
must be committed to the new branch, and have its JIRA version tagged
appropriately.

All changes to be cherrypicked into 3.3.5, except for those ones related to
the release process itself, MUST be in branch-3.3 first, and SHOULD be in
trunk unless there is some fundamental reason they can't apply there
(reload4j etc).

Let's try and stabilise this releases, especially bringing up to date all
the JAR dependencies which we can safely update.

Anyone planning to give talks at ApacheCon about forthcoming features
already in 3.3 SHOULD

   1. reference Hadoop 3.3.5 as the version
   2. make sure their stuff works.

Mukund will be at the conf; find him and offer any help you can in getting
this release out.

I'd like to get that Arm64 build workingdoes anyone else want to get
involved?

-steve

the next hadoop 3.3.x release

[Steve Loughran]

Hi

Mukund Thakur plans to fork off the next branch-3.3. release in 10 days
time. last chance to get changes in.

I'm away next week...try not to break the branch. Any testing you can do
would be appreciated

Steve

[jira] [Resolved] (HDFS-16755) TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16755.
---
Fix Version/s: 3.3.9
   Resolution: Fixed

> TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host 
> resolution
> ---
>
> Key: HDFS-16755
> URL: https://issues.apache.org/jira/browse/HDFS-16755
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: 3.4.0, 3.3.9
> Environment: Running using both Maven Surefire and an IDE results in 
> a test failure.  Switching the name to "bogus.invalid" results in the 
> expected behavior, which depends on an UnknownHostException.
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Tests that want to use an unresolvable address may actually resolve in some 
> environments.  Replacing host names like "bogus" with a IETF RFC 2606 domain 
> name avoids the issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Protobuf-java compatibilty question

[Steve Loughran]

I'm afraid not. protobuf maybe compatible at the wire level -the marshalled
data is good-, but the generated classes in jars Will not link to any
version other than that they were explicitly compiled with.
this is why hadoop 3.3 has a private shaded copy in the hadoop-thirdparty
JAR -how to stop the Hadoop/hdfs code restricting what others can use.

On Tue, 9 Aug 2022 at 16:51, Jason Wen 
wrote:

> Hi team,
>
> Hadooop 3.2.x release is using protobuf-java 2.5.0 version lib. It looks
> like protobuf-java 3.x is binary backward compatible with 2.5.0 version.
> Does that mean Hadoop 3.2.x can work correctly if we replace protobuf-java
> 2.5.0 lib with 3.x version at runtime?
>
> Thanks,
> Jason
>

[ANNOUNCE] Apache Hadoop 3.3.4 release

[Steve Loughran]

On behalf of the Apache Hadoop Project Management Committee, I am
pleased to announce the release of Apache Hadoop 3.3.4.

---
This is a release of Apache Hadoop 3.3 line.

It contains a small number of security and critical integration fixes since
3.3.3.

Users of Apache Hadoop 3.3.3 should upgrade to this release.

Users of hadoop 2.x and hadoop 3.2 should also upgrade to the 3.3.x line.
As well as feature enhancements, this is the sole branch currently
receiving fixes for anything other than critical security/data integrity
issues.

Users are encouraged to read the [overview of major changes][1] since
release 3.3.3.
For details of bug fixes, improvements, and other enhancements since the
previous 3.3.3 release,
please check [release notes][2] and [changelog][3].

[1]: http://hadoop.apache.org/docs/r3.3.4/index.html
[2]:
http://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-common/release/3.3.4/RELEASENOTES.3.3.4.html
[3]:
http://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-common/release/3.3.4/CHANGELOG.3.3.4.html


Many thanks to everyone who helped in this release by supplying patches,
reviewing them, helping get this release building and testing and
reviewing the final artifacts.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

thanks.

the release is up for download as is the site; i will do the announcement
now.

also automated a lot more of the work of doing the release, inc testing
across projects
https://github.com/steveloughran/validate-hadoop-client-artifacts

On Thu, 4 Aug 2022 at 19:08, Stack  wrote:

> +1 (Sorry, took me a while)
>
> Ran: ./dev-support/hadoop-vote.sh --source
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>
> * Signature: ok
>
> * Checksum : failed
>
> * Rat check (17.0.1): ok
>
>  - mvn clean apache-rat:check
>
> * Built from source (17.0.1): ok
>
>  - mvn clean install  -DskipTests
>
> * Built tar from source (17.0.1): ok
>
>  - mvn clean package  -Pdist -DskipTests -Dtar
> -Dmaven.javadoc.skip=true
>
> Took a look at website. Home page says stuff like, “ARM Support: This is
> the first release to support ARM architectures.“, which I don’t think is
> true of 3.3.4 but otherwise, looks fine.
>
> Only played with HDFS. UIs looked right.
>
> Deployed to ten node arm64 cluster. Ran the hbase verification job on top
> of it and all passed. Did some kills, stuff came back.
>
> I didn't spend time on unit tests but one set passed on a local rig here:
>
> [image: image.png]
> Stack
>
> On Fri, Jul 29, 2022 at 11:48 AM Steve Loughran
>  wrote:
>
>> I have put together a release candidate (RC1) for Hadoop 3.3.4
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>>
>> The git tag is release-3.3.4-RC1, commit a585a73c3e0
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1358/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
>>
>> Release notes
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
>>
>> There's a very small number of changes, primarily critical code/packaging
>> issues and security fixes.
>>
>> See the release notes for details.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> steve
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

The vote passed with the following result.

Binding PMC members:

+1 Chris Nauroth
+1 Steve Loughran
+1 Masatake Iwasaki

Non binding votes:

+1 Ashutosh Gupta

Cheng Pan was worried about the new transient kotlin dependency. They are
working on a PR there which we can target for the successor to this release
https://issues.apache.org/jira/browse/HDFS-16714

I'm going to publish the artifacts, site docs, maven artifacts, then
announce.

Thank you to all who helped to push this release out.


On Thu, 4 Aug 2022 at 11:48, Ashutosh Gupta 
wrote:

> +1 (non-binding)
>
> * Builds from source looks good.
> * Checksums and signatures are correct.
> * Running basic HDFS commands and running simple MapReduce jobs looks good.
> * Skimmed through the contents of site documentation and it looks good.
>
> Thanks Steve for driving this release.
>
> Ashutosh
>
>
> On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth  wrote:
>
> > +1 (binding)
> >
> > * Verified all checksums.
> > * Verified all signatures.
> > * Built from source, including native code on Linux.
> > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> > -Drequire.zstd -DskipTests
> > * Tests passed.
> > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > * Checked dependency tree to make sure we have all of the expected
> library
> > updates that are mentioned in the release notes.
> > * mvn -o dependency:tree
> >
> > I saw a LibHDFS test failure, but I know it's something flaky that's
> > already tracked in a JIRA issue. The release looks good. Steve, thank you
> > for driving this.
> >
> > Chris Nauroth
> >
> >
> > On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran
>  > >
> > wrote:
> >
> > > my vote for this is +1, binding.
> > >
> > > obviously I`m biased, but i do not want to have to issue any more
> interim
> > > releases before the feature release off branch-3.3, so I am trying to
> be
> > > ruthless.
> > >
> > > my client vaidator ant project has a more targets to help with
> releasing,
> > > and now builds a lot mor of my local projects
> > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > > all good as far as my test coverage goes, with these projects
> validating
> > > the staged dependencies.
> > >
> > > now, who else can review
> > >
> > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran 
> > wrote:
> > >
> > > >
> > > >
> > > > I have put together a release candidate (RC1) for Hadoop 3.3.4
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
> > > >
> > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1358/
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
> > > >
> > > > There's a very small number of changes, primarily critical
> > code/packaging
> > > > issues and security fixes.
> > > >
> > > > See the release notes for details.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > steve
> > > >
> > >
> >
>

[jira] [Resolved] (HDFS-16711) Empty hadoop-client-api artifacts on maven central

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16711.
---
Resolution: Won't Fix

closing as "cantfix". sorry

> Empty hadoop-client-api artifacts on maven central
> --
>
> Key: HDFS-16711
> URL: https://issues.apache.org/jira/browse/HDFS-16711
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.2.3
>Reporter: Robin Wolters
>Priority: Major
>
> I observed that for at least version 3.2.3 the artifacts on maven central for 
> the shaded jars of both 
> [hadoop-client-api|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.2.3/]
>  and 
> [hadoop-client-runtime|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.2.3/]
>  are empty, i.e. of ~45KB size and do not contain any class files if 
> extracted (or listed with "jar tf").
> I've come across this [e-mail 
> thread|https://www.mail-archive.com/common-dev@hadoop.apache.org/msg37261.html]
>  suggesting that there was the same problem with version 3.3.3, which appears 
> to be fixed. Version 3.2.3 is mentioned as well, could it be that this 
> version simply wasn't re-released?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

my vote for this is +1, binding.

obviously I`m biased, but i do not want to have to issue any more interim
releases before the feature release off branch-3.3, so I am trying to be
ruthless.

my client vaidator ant project has a more targets to help with releasing,
and now builds a lot mor of my local projects
https://github.com/steveloughran/validate-hadoop-client-artifacts
all good as far as my test coverage goes, with these projects validating
the staged dependencies.

now, who else can review

On Fri, 29 Jul 2022 at 19:47, Steve Loughran  wrote:

>
>
> I have put together a release candidate (RC1) for Hadoop 3.3.4
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>
> The git tag is release-3.3.4-RC1, commit a585a73c3e0
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1358/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
> See the release notes for details.
>
> Please try the release and vote. The vote will run for 5 days.
>
> steve
>

[VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

I have put together a release candidate (RC1) for Hadoop 3.3.4

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/

The git tag is release-3.3.4-RC1, commit a585a73c3e0

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1358/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

See the release notes for details.

Please try the release and vote. The vote will run for 5 days.

steve

any HDFS blockers for a 3.3.x feature release?

[Steve Loughran]

Is there any WiP which people consider a blocker for a  3.3.x feature
release to follow the 3.3.4 security/critical integration issues release?

I want to split off the fork for that next release off branch-3.3 and start
stabilising.

are there things people working on which are nearly ready?

[jira] [Resolved] (HDFS-9475) execution of org.apache.hadoop.hdfs.net.TcpPeerServer.close() causes timeout on Hadoop-2.6.0 with IBM-JDK-1.8

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-9475.
--
Resolution: Won't Fix

> execution of org.apache.hadoop.hdfs.net.TcpPeerServer.close() causes timeout 
> on Hadoop-2.6.0 with IBM-JDK-1.8
> -
>
> Key: HDFS-9475
> URL: https://issues.apache.org/jira/browse/HDFS-9475
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.1
> Environment: IBM JDK 1.8.0
> Architecture:  s390x GNU/Linux
>Reporter: Rakesh Sharma
>Priority: Blocker
>
> ---
> Test set: org.apache.hadoop.hdfs.server.balancer.TestBalancer
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.69 sec 
> <<< FAILURE! - in org.apache.hadoop.hdfs.server.balancer.TestBalancer
> testTwoReplicaShouldNotInSameDN(org.apache.hadoop.hdfs.server.balancer.TestBalancer)
>   Time elapsed: 100.008 sec  <<< ERROR!
> java.lang.Exception: test timed out after 10 milliseconds
>   at 
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:245)
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:126)
>   at sun.nio.ch.ServerSocketAdaptor.close(ServerSocketAdaptor.java:149)
>   at 
> org.apache.hadoop.hdfs.net.TcpPeerServer.close(TcpPeerServer.java:153)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.kill(DataXceiverServer.java:223)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:1663)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1750)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1721)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
>   at 
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.testTwoReplicaShouldNotInSameDN(TestBalancer.java:1382)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

cancelling this RC so i can issue a new one with an updated reload4j
https://issues.apache.org/jira/browse/HADOOP-18354

i will also do a pr to update aws sdk, which is less critical (the jackson
databind classes don't seem to get used), -upgrading will stop audit tools
overreacting

On Thu, 21 Jul 2022 at 19:07, Steve Loughran  wrote:

>
> I have put together a release candidate (RC0) for Hadoop 3.3.4
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/
>
> The git tag is release-3.3.4-RC0, commit c679bc76d26
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1356/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
> See the release notes for details.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>
>
>

[jira] [Created] (HDFS-16679) StandbyState link

[Steve Loughran (Jira)]

Steve Loughran created HDFS-16679:
-

 Summary: StandbyState link
 Key: HDFS-16679
 URL: https://issues.apache.org/jira/browse/HDFS-16679
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Steve Loughran






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

I have put together a release candidate (RC0) for Hadoop 3.3.4

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/

The git tag is release-3.3.4-RC0, commit c679bc76d26

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1356/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

See the release notes for details.

Please try the release and vote. The vote will run for 5 days.

-Steve

[jira] [Resolved] (HDFS-16651) io.netty:netty CVE-2019-20444, CVE-2019-20445

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16651.
---
Resolution: Duplicate

there's some outstanding netty upgrade jiras; remember to check before creating 
new ones. 

updating any jar is often traumatic; any help in testing is always welcome

> io.netty:netty CVE-2019-20444, CVE-2019-20445
> -
>
> Key: HDFS-16651
> URL: https://issues.apache.org/jira/browse/HDFS-16651
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: 3.1.1
>Affects Versions: 3.3.3
>Reporter: Basavaraj
>Priority: Critical
> Attachments: netty_vuln.html
>
>
> netty library has security issues. Need to upgrade all netty-all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Hadoop 3.3.4 release process underway

[Steve Loughran]

update


   1. branch 3.3 has a version of 3.3.9-SNAPSHOT
   2. and 3.3.9 should be the version for fixes against this, with 3.3.4
   for the new point release
   3. please don't use 3.3.4 for branch-3.3 changes from now on

I've already got a PR up with the changes; going to create an asf
branch-3.3.4 branch mirroring it and kicking off with the pom update.

I'm going to do a dry run of a release this week to build the binaries on
x86 and ARM but not put for a vote as I am away next week. instead it'll be
a validation of my processes, ant-based automation etc. (
https://github.com/steveloughran/validate-hadoop-client-artifacts)

 I will kick off the release the following week, which, being july4 week,
may have more people offline. it does give larry a chance to get
https://issues.apache.org/jira/browse/HADOOP-18074 in, as it may have
security implications.



On Mon, 20 Jun 2022 at 19:02, Steve Loughran  wrote:

> I'm setting things up for a new release
>
> https://issues.apache.org/jira/browse/HADOOP-18305
>
> absolute minimum of fixes. as well as some related to ZK lockdown, i would
> like to include
>
> https://issues.apache.org/jira/browse/HADOOP-18303
> remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
>
> and
> https://issues.apache.org/jira/browse/HADOOP-18307
> remove hadoop-cos as a dependency of hadoop-cloud-storage
>
> the last one is a lstminute workaround for, a classpath ordering issue due
> to too many libraries having unshaded references to mozilla/prefix-list.txt
> https://issues.apache.org/jira/browse/HADOOP-18159
>
> the proper fixes would be getting that updated library tested (who is set
> up to test tencent cloud?) and ideally (aws, cos, gcs shaded libraries to
> shade their references)
>
> for 3.3.4, i am just going to cut the declaration of the module as a
> dependency of hadoop-cloud-storage so out of downstream apps unless they
> explicitly ask forit.
>
> now, numbering
>
>
>1. I am calling this 3.3.4
>2. I am going to increase the version of branch 3.3. to 3.3.9. that
>leaves space for some more but doesn't confuse jira dropdown dialogs.
>
>
> i do believe branch-3.3. should be renamed branch-3.4 and the release i
> plan to do with mukund called 3.4.0, but that is another bit of project
> organisation.
>
> expect the first RC up soon, I am going to be away on vacation from june
> 28 to july 23 though, which complicates things
>
>
>

Hadoop 3.3.4 release process underway

[Steve Loughran]

I'm setting things up for a new release

https://issues.apache.org/jira/browse/HADOOP-18305

absolute minimum of fixes. as well as some related to ZK lockdown, i would
like to include

https://issues.apache.org/jira/browse/HADOOP-18303
remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime

and
https://issues.apache.org/jira/browse/HADOOP-18307
remove hadoop-cos as a dependency of hadoop-cloud-storage

the last one is a lstminute workaround for, a classpath ordering issue due
to too many libraries having unshaded references to mozilla/prefix-list.txt
https://issues.apache.org/jira/browse/HADOOP-18159

the proper fixes would be getting that updated library tested (who is set
up to test tencent cloud?) and ideally (aws, cos, gcs shaded libraries to
shade their references)

for 3.3.4, i am just going to cut the declaration of the module as a
dependency of hadoop-cloud-storage so out of downstream apps unless they
explicitly ask forit.

now, numbering


   1. I am calling this 3.3.4
   2. I am going to increase the version of branch 3.3. to 3.3.9. that
   leaves space for some more but doesn't confuse jira dropdown dialogs.


i do believe branch-3.3. should be renamed branch-3.4 and the release i
plan to do with mukund called 3.4.0, but that is another bit of project
organisation.

expect the first RC up soon, I am going to be away on vacation from june 28
to july 23 though, which complicates things

[DISCUSS] Filesystem API shim library to assist applications still targeting previous hadoop releases.

[Steve Loughran]

I've just created an initial project "fs-api-shim" to provide controlled
access to the hadoop 3.3.3+ filesystem API calls on hadoop 3.2.0+ releases
https://github.com/steveloughran/fs-api-shim

The goal here is to make it possible for core file format libraries
(Parquet, Avro, ORC, Arrow etc) and other apps (HBase, ...) to take
advantage of those APIs which we have updated and optimised for access to
cloud stores. Currently the applications do not and are under performance
on recent releases. I have the ability to change our internal forks but I
would like to let others gain from the changes and avoid having to diverge
i'll internal libraries too much.

Currently too many libraries seen frozen in time

Avro: still rejecting changes which don't compile on hadoop 2
https://github.com/apache/avro/pull/1431

Parquet: still using reflection to access non hadoop 1.x filesystem API
calls
https://github.com/apache/parquet-mr/pull/971

I'm not going to support hadoop 2.10 —but we can at least say "move up to
hadoop 3.2.x and we will let you use later APIs when available"

some calls, like openFile() will work everywhere; on versions with the open
file builder API they will take the final status and fake policy so let
libraries declare whether they are random/sequential is IO and skip those
HEAD requests on the object stores they do to verify that the file exists
and determine its length for the ranged GET call requests which will follow.

https://github.com/steveloughran/fs-api-shim/blob/main/fs-api-shim-library/src/main/java/org/apache/hadoop/fs/shim/FileSystemShim.java#L38

On Hadoop 3.2.x, or if openFile() fails for some reason, it will just
downgrade to the classic open() call.

Other API calls we can support dynamic binding to through reflection but
not actually fallback if they are unavailable. This will allow libraries to
use the API calls if present but force them to come up with alternative
solutions if not.

A key part of this is FSDataInputStream, where the ByteBufferReadable API
would be benefit to Parquet

https://github.com/steveloughran/fs-api-shim/blob/main/fs-api-shim-library/src/main/java/org/apache/hadoop/fs/shim/FSDataInputStreamShim.java

When we get the vectored IO feature branch in, we can offer similar
reflection-based access. It means applications can compile on hadoop 3.2.x
and 3.3.x but still take advantage of the APIs when they are on a version
without it.

I'm going to stay clear of more complicated APIs which don't offer tangible
performance gains and which are very hard to do (IOStatistics).

Testing is fun; I have a plan there which consists of FS contract tests in
the shim test source tree to verify the 3.2.0 functionality and an adjacent
module which will run those same tests against more recent versions. I need
test will have to beat targetable against objects doors as well as local
and mini HGFS for systems

This is all in github; however it is very much a hadoop extension library.
Is there a way we could release it as an ASF Library but on a different
timetable from normal Hadoop releases? There is always incubator, but this
is such a minor project it is closer to the org.apache.hadoop.thirdparty
library in that it is something all current committers okay should be able
to commit to and release, while releasing on a schedule independent of
hadoop releases themselves. Having it come from this project should give it
more legitimacy.

Steve

[DISCUSS] Forthcoming Hadoop releases

[Steve Loughran]

I want to start a quick discussion on a plan for hadoop releases this
summer. I am willing to do the release manager work. Mukund and Mehakmeet
have have already volunteered to help even if they don't know that yet.

I've got two goals

   1. minor followup to 3.3.3
   2. feature release of new stuff


*Followup to 3.3.3, working title "3.3.4"*

I've a PR up on github to add those change to the 3.3.2/3.3.3 line which
have shipped elsewhere and/or we consider critical.

https://github.com/apache/hadoop/pull/4345

This is for critical data integrity/service availability patches; things
like test values we will just triage.

I can start a new release of this at the end of the week, with an RC up
next week ready for review. With the wonderful docker based build and some
extra automation I've been adding for validating releases
(validate-hadoop-client-artifacts), getting that RC out is not that
problematic; issuing git commands is the heavy lifting.

What does take effort is the testing by everybody else; the smaller the set
of changes the more this is limited to validating the artifacts and the
maven publishing.

As it is a follow up to hadoop 3.3.3 then it needs the version number
3.3.4. This raises the question "what about branch-3.3", which brings me to
the next deliverable.

*branch-3.3 => branch-3.4, targeting hadoop 3.4.0 in 3Q22*

With the 3.3.x line being maintained for critical fixes only, make the
hadoop version in branch-3.3 "hadoop-3.4.0" and release later this year.

A release schedule which is probably doable despite people taking time off
over the summer could be

   - feature complete by July/August
   - RC(s) sept/oct with goal of shipping by October


I volunteer to be release manager, albeit with critical help from
colleagues. For people who haven't worked with me on a project release
before, know that I'm fairly ruthless about getting changes in once the
branch is locked down. So get those features in now.

hadoop trunk gets its version number incremented to 3.5.0-SNAPSHOT

It's probably time we think about what a release off trunk would mean -but
t I would like to get a branch-3.3 release out rather than later.

What do people think of this? And is there anyone else willing to get
involved with the release process?

-Steve

[jira] [Resolved] (HDFS-16563) Namenode WebUI prints sensitive information on Token Expiry

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16563.
---
Resolution: Fixed

> Namenode WebUI prints sensitive information on Token Expiry
> ---
>
> Key: HDFS-16563
> URL: https://issues.apache.org/jira/browse/HDFS-16563
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namanode, security, webhdfs
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
> Attachments: image-2022-04-27-23-01-16-033.png, 
> image-2022-04-27-23-28-40-568.png
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Login to Namenode WebUI.
> Wait for token to expire. (Or modify the Token refresh time 
> dfs.namenode.delegation.token.renew/update-interval to lower value)
> Refresh the WebUI after the Token expiry.
> Full token information gets printed in WebUI.
>  
> !image-2022-04-27-23-01-16-033.png!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-16563) Namenode WebUI prints sensitive information on Token Expiry

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HDFS-16563:
---

> Namenode WebUI prints sensitive information on Token Expiry
> ---
>
> Key: HDFS-16563
> URL: https://issues.apache.org/jira/browse/HDFS-16563
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namanode, security, webhdfs
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
> Attachments: image-2022-04-27-23-01-16-033.png, 
> image-2022-04-27-23-28-40-568.png
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Login to Namenode WebUI.
> Wait for token to expire. (Or modify the Token refresh time 
> dfs.namenode.delegation.token.renew/update-interval to lower value)
> Refresh the WebUI after the Token expiry.
> Full token information gets printed in WebUI.
>  
> !image-2022-04-27-23-01-16-033.png!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Steve Loughran]

+1 binding

I've extended my validator project
https://github.com/steveloughran/validate-hadoop-client-artifacts

it now
* fetches KEYS
* fetches an RC from a remote url
* validates signing
* untars and tries to build source
* untars binary release and tries some native commands

the source build fails because I'm on a mac M1 and branch-2 doesn't have
the dynamic protoc switching. not sure if it can go in at all.

binary untar worked, basic commands ok except for bin/hadoop checknative
-again, no native libs for this system.

so: as far as all my newly automated process goes, and allowing for mac m1
native code issues, I'm happy.

On Wed, 25 May 2022 at 03:41, Masatake Iwasaki 
wrote:

> Hi all,
>
> Here's Hadoop 2.10.2 release candidate #0:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1350
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[ANNOUNCE] Apache Hadoop 3.3.3 release

[Steve Loughran]

On behalf of the Apache Hadoop Project Management Committee, I'm pleased to
announce that Hadoop 3.3.3 has been
released:

https://hadoop.apache.org/release/3.3.3.html

This is the third stable release of the Apache Hadoop 3.3 line.

It contains 23 bug fixes, improvements and enhancements since 3.3.2.

This is primarily a security update; for this reason, upgrading is strongly
advised.

Users are encouraged to read the overview of major changes[1] since 3.3.2.
For details of bug fixes, improvements, and other enhancements since the
previous 3.3.2 release,
please check release notes[2] and changelog[3].

[1]: /docs/r3.3.3/index.html
[2]:
http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/RELEASENOTES.3.3.3.html
[3]:
http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/CHANGELOG.3.3.3.html


As the release notes highlight, this release contains HADOOP-18088 "Replace
log4j 1.x with reload4j"
https://issues.apache.org/jira/browse/HADOOP-18088

This ensures that the version of log4j shipped is free of known CVEs. the
standard log4j 1.2.17 has some known CVEs in classes which were never uses;
reload4j cuts them out. Audit scanning tools should stop highlighting
perceived risks here.

If you are using maven exclusions to manage logging libraries, or were
otherwise replacing the log4j artifacts in deployments, note the different
library/artifact names which need to be handled.

Many thanks to everyone who helped in this release by supplying patches,
reviewing them, helping get this release building and testing reviewing the
final artifacts.

Steve

Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Steve Loughran]

Here are the result of the vote.

Binding PMC members

+1 Stack
+1 Masatake Iwasaki
+1 Xiaoqiao He
+1 Chao Sun
+1 Chris Nauroth


Non binding votes

+1 Ayush Saxena
+1 Viraj Jasani
+1 Mukund Thakur


There were no negative votes, and maven is happy.
I'm pleased to announce therefore, that this release candidate has been
approved to become the final release.

I'm going to go through the next steps of the process: publishing the
artifacts, site docs, maven artifacts, announcements etc.

Thank you to all who reviewed this!

-Steve

PS: I'm going to start some discussion on correcting/tuning the release
process, once I've got through it.

On Mon, 16 May 2022 at 19:16, Chris Nauroth  wrote:

> +1 (binding)
>
> - Verified all checksums.
> - Verified all signatures.
> - Built from source, including native code on Linux.
> - Ran several examples successfully.
>
> Chris Nauroth
>
>
> On Mon, May 16, 2022 at 10:06 AM Chao Sun  wrote:
>
> > +1
> >
> > - Compiled from source
> > - Verified checksums & signatures
> > - Launched a pseudo HDFS cluster and ran some simple commands
> > - Ran full Spark tests with the RC
> >
> > Thanks Steve!
> >
> > Chao
> >
> > On Mon, May 16, 2022 at 2:19 AM Ayush Saxena  wrote:
> > >
> > > +1,
> > > * Built from source.
> > > * Successful native build on Ubuntu 18.04
> > > * Verified Checksums.
> > >
> >
> (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
> > > * Verified Signature.
> > > * Successful RAT check
> > > * Ran basic HDFS shell commands.
> > > * Ran basic YARN shell commands.
> > > * Verified version in hadoop version command and UI
> > > * Ran some MR example Jobs.
> > > * Browsed
> UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer)
> > > * Browsed the contents of Maven Artifacts.
> > > * Browsed the contents of the website.
> > >
> > > Thanx Steve for driving the release, Good Luck!!!
> > >
> > > -Ayush
> > >
> > > On Mon, 16 May 2022 at 08:20, Xiaoqiao He 
> wrote:
> > >
> > > > +1(binding)
> > > >
> > > > * Verified signature and checksum of the source tarball.
> > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean
> package
> > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > * Setup pseudo cluster with HDFS and YARN.
> > > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result.
> > > > * Run example mr applications and check the result - Pi & wordcount.
> > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> > etc.
> > > >
> > > > Thanks Steve for your work.
> > > >
> > > > - He Xiaoqiao
> > > >
> > > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani 
> > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * Signature: ok
> > > > > * Checksum : ok
> > > > > * Rat check (1.8.0_301): ok
> > > > >  - mvn clean apache-rat:check
> > > > > * Built from source (1.8.0_301): ok
> > > > >  - mvn clean install  -DskipTests
> > > > > * Built tar from source (1.8.0_301): ok
> > > > >  - mvn clean package  -Pdist -DskipTests -Dtar
> > -Dmaven.javadoc.skip=true
> > > > >
> > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on
> > > > > pseudo-distributed mode looks good.
> > > > >
> > > > >
> > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran
> > > > 
> > > > > wrote:
> > > > >
> > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3
> > > > > >
> > > > > > The RC is available at:
> > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/
> > > > > >
> > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3
> > > > > >
> > > > > > The maven artifacts are staged at
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1349/
> > > > > >
> > > > > > You can find my public key at:
> > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > > >
> > > > > 

[VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Steve Loughran]

I have put together a release candidate (RC1) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/

The git tag is release-3.3.3-RC1, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1349/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

* The critical fixes which shipped in the 3.2.3 release.
* CVEs in our code and dependencies
* Shaded client packaging issues.
* A switch from log4j to reload4j

reload4j is an active fork of the log4j 1.17 library with the classes
which contain CVEs removed. Even though hadoop never used those classes,
they regularly raised alerts on security scans and concen from users.
Switching to the forked project allows us to ship a secure logging
framework. It will complicate the builds of downstream
maven/ivy/gradle projects which exclude our log4j artifacts, as they
need to cut the new dependency instead/as well.

See the release notes for details.

This is the second release attempt. It is the same git commit as before, but
fully recompiled with another republish to maven staging, which has bee
verified by building spark, as well as a minimal test project.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran  wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun  wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>> 
>>staged
>>staged-releases
>>https://repository.apache.org/content/repositories/staging/
>> 
>>
>>      true
>>
>>
>>  true
>>
>>  
>>
>> to the `` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>>  wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena 
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > >>>
>> > > >>> Hm

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun  wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
> 
>staged
>staged-releases
>https://repository.apache.org/content/repositories/staging/
> 
>
>  true
>
>
>  true
>
>  
>
> to the `` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>  wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena 
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena 
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun 
> wrote:
> > > >>>>>
> > > >>&

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki 
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena  wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena 
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun  wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>- mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>- mvn clean install  -DskipTests
> >>>>>>- mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>> - HDFS Tests passed (Didn't run more than 

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Steve Loughran]

how about for trunk we -wait for it- declare it java11 only?

do that and we cut out a lot of the CI build.

we would have to mandate a test run through branch-3.3 before any
cherrypick there

On Sat, 7 May 2022 at 11:19, Ayush Saxena  wrote:

> Three for trunk:
> Java-8
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/
>
> Java-11
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/
>
> ARM
>
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/
>
> -Ayush
>
> >
> > On 07-May-2022, at 11:49 AM, Gautham Banasandra 
> wrote:
> >
> > Hi all,
> >
> > Although validating cross platform builds at pre-commit
> > would be the most effective approach, I understand the
> > huge disadvantage caused by the slowdown. The best
> > way to tackle this would be to enable parallel builds for
> > the different platforms. I had given it a shot about a year
> > ago[1], it didn't go well and ran into all sorts of random
> > errors. I think we should make the parallel builds run on
> > different agents as opposed to starting the builds parallelly
> > on the same agent (which is what I had done earlier).
> >
> > So, I'll settle down to integrating the full suite of platform
> > builds into the nightly builds. Could anyone please point
> > me to the Jenkins job for this?
> >
> > [1] = https://github.com/apache/hadoop/pull/3166
> >
> > Thanks,
> > --Gautham
> >
> >> On Fri, 6 May 2022 at 21:04, Ayush Saxena  wrote:
> >>
> >> From functional point of view it does makes sense to validate all the
> >> platforms as part of the builds, but the Pre commits builds taking time
> is
> >> now no longer a small things, In past one year or may be two, we have
> >> already increased it more than twice as compared to what it was before,
> If
> >> someone has a change in HDFS, which includes both hdfs-client &
> >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2
> hours.
> >> With the current state, I don't think we should just go and add these
> >> extra overheads. Having them as part of the nightly builds does makes
> sense
> >> for now.
> >>
> >> In future if we feel there is a strong need for this and we start to see
> >> very frequent failures in some other platforms and we are left with no
> >> other option but to integrate it in our pre-commit jobs, we can explore
> >> having these build phases running in parallel, along with trying other
> >> phases also to run in parallel like compilation/javadoc builds of JDK-8
> &
> >> JDK-11 can run in parallel and may be explore other opportunities as
> well
> >> to compensate for this time.
> >>
> >> For now lets just integrate it our nightly builds only and circle back
> >> again here in future if the need be.
> >>
> >> -Ayush
> >>
> >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang 
> wrote:
> >>>
> >>> Running builds for all platforms for each and every PR seems too
> >>> excessive.
> >>>
> >>> How about doing all platform builds in the nightly jobs?
> >>>
> >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran
>  >>>>
> >>> wrote:
> >>>
> >>>> I'm not enthusiastic here as it not only makes the builds slower, it
> >>>> reduces the #of builds we can through a day
> >>>>
> >>>> one thing I am wondering is could we remove java8 support on some
> >>> branches?
> >>>>
> >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and
> >>> this
> >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as java
> 11
> >>>> only.
> >>>> that would cut buld and test time for those trunk PRs in half...after
> >>> which
> >>>> the preospect of building on more than one platform becomes more
> viable.
> >>>>
> >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra 
> >>>> wrote:
> >>>>
> >>>>> Hi Hadoop devs,
> >>>>>
> >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by
> >>>>> https://github.com/apache/hadoop/pull/3988. In
> >>> dev-support/jenkins.sh,
> >>>>> there's the capability to build and test Hadoop across the supported
> >>>>> platform

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

t;> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>- HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > 
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing  to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>>
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Steve Loughran]

I'm not enthusiastic here as it not only makes the builds slower, it
reduces the #of builds we can through a day

one thing I am wondering is could we remove java8 support on some branches?

make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and this
summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11
only.
that would cut buld and test time for those trunk PRs in half...after which
the preospect of building on more than one platform becomes more viable.

On Thu, 5 May 2022 at 15:34, Gautham Banasandra  wrote:

> Hi Hadoop devs,
>
> Last week, there was a Hadoop build failure on Debian 10 caused by
> https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh,
> there's the capability to build and test Hadoop across the supported
> platforms. Currently, we're limiting this only for those PRs having only
> C/C++ changes[1], since C/C++ changes are more likely to cause
> cross-platform build issues and bypassing the full platform build for non
> C/C++ PRs would save a great deal of CI time. However, the build failure
> caused by PR #3988 motivates me to enable the capability to build and
> test Hadoop for all the supported platforms for ALL the PRs.
>
> While this may cause longer CI run duration for each PR, it would
> immensely minimize the risk of breaking Hadoop across platforms and
> saves us a lot of debugging time. Kindly post your opinion regarding this
> and I'll move to enable this capability for all PRs if the response is
> sufficiently positive.
>
> [1] =
>
> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103
>
>
> Thanks,
> --Gautham
>

[VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I have put together a release candidate (rc0) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/

The git tag is release-3.3.3-RC0, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1348/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.


   - The critical fixes which shipped in the 3.2.3 release.
   -  CVEs in our code and dependencies
   - Shaded client packaging issues.
   - A switch from log4j to reload4j


reload4j is an active fork of the log4j 1.17 library with the classes which
contain CVEs removed. Even though hadoop never used those classes, they
regularly raised alerts on security scans and concen from users. Switching
to the forked project allows us to ship a secure logging framework. It will
complicate the builds of downstream maven/ivy/gradle projects which exclude
our log4j artifacts, as they need to cut the new dependency instead/as well.

See the release notes for details.

This is my first release through the new docker build process, do please
validate artifact signing  to make sure it is good. I'll be trying builds
of downstream projects.

We know there are some outstanding issues with at least one library we are
shipping (okhttp), but I don't want to hold this release up for it. If the
docker based release process works smoothly enough we can do a followup
security release in a few weeks.

Please try the release and vote. The vote will run for 5 days.

-Steve

[jira] [Resolved] (HDFS-14478) Add libhdfs APIs for openFile

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-14478.
---
Fix Version/s: 3.3.4
   Resolution: Fixed

> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>    Assignee: Steve Loughran
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.4
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows 
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and 
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
>   int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs 
> would just expose the functionality of {{Future}}.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16501) Print the exception when reporting a bad block

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16501.
---
Resolution: Fixed

> Print the exception when reporting a bad block
> --
>
> Key: HDFS-16501
> URL: https://issues.apache.org/jira/browse/HDFS-16501
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.3
>
> Attachments: image-2022-03-10-19-27-31-622.png
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> !image-2022-03-10-19-27-31-622.png|width=847,height=27!
> Currently, volumeScanner will find bad block and report it to namenode 
> without printing the reason why the block is a bad block. I think we should 
> be better print the exception in log file.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16437) ReverseXML processor doesn't accept XML files without the SnapshotDiffSection.

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16437.
---
Resolution: Fixed

> ReverseXML processor doesn't accept XML files without the SnapshotDiffSection.
> --
>
> Key: HDFS-16437
> URL: https://issues.apache.org/jira/browse/HDFS-16437
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.1.1, 3.3.0
>Reporter: yanbin.zhang
>Assignee: yanbin.zhang
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.3.3, 3.2.3
>
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> In a cluster environment without snapshot, if you want to convert back to 
> fsimage through the generated xml, an error will be reported.
> {code:java}
> //代码占位符
> [test@test001 ~]$ hdfs oiv -p ReverseXML -i fsimage_0257220.xml 
> -o fsimage_0257220
> OfflineImageReconstructor failed: FSImage XML ended prematurely, without 
> including section(s) SnapshotDiffSection
> java.io.IOException: FSImage XML ended prematurely, without including 
> section(s) SnapshotDiffSection
>         at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.processXml(OfflineImageReconstructor.java:1765)
>         at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageReconstructor.run(OfflineImageReconstructor.java:1842)
>         at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:211)
>         at 
> org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:149)
> 22/01/25 15:56:52 INFO util.ExitUtil: Exiting with status 1: ExitException 
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16355) Improve the description of dfs.block.scanner.volume.bytes.per.second

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16355.
---
Fix Version/s: 3.3.3
   (was: 3.4.0)
   (was: 3.3.4)
   Resolution: Fixed

FIxed in 3.3.3; updating fix versions as appropriate

> Improve the description of dfs.block.scanner.volume.bytes.per.second
> 
>
> Key: HDFS-16355
> URL: https://issues.apache.org/jira/browse/HDFS-16355
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, hdfs
>Affects Versions: 3.3.1
>Reporter: guophilipse
>Assignee: guophilipse
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> datanode block scanner will be disabled if 
> `dfs.block.scanner.volume.bytes.per.second` is configured less then or equal 
> to zero, we can improve the desciption



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11041) Unable to unregister FsDatasetState MBean if DataNode is shutdown twice

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-11041.
---
Fix Version/s: 3.3.3
   (was: 3.4.0)
   (was: 3.3.4)
   Resolution: Fixed

FIxed in 3.3.3; updating fix versions as appropriate

> Unable to unregister FsDatasetState MBean if DataNode is shutdown twice
> ---
>
> Key: HDFS-11041
> URL: https://issues.apache.org/jira/browse/HDFS-11041
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Fix For: 2.10.2, 3.3.3, 3.2.3
>
> Attachments: HDFS-11041.01.patch, HDFS-11041.02.patch, 
> HDFS-11041.03.patch
>
>
> I saw error message like the following in some tests
> {noformat}
> 2016-10-21 04:09:03,900 [main] WARN  util.MBeans 
> (MBeans.java:unregister(114)) - Error unregistering 
> Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
> javax.management.InstanceNotFoundException: 
> Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
>   at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
>   at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
>   at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:112)
>   at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:2127)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2016)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1985)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
>   at 
> org.apache.hadoop.hdfs.TestDatanodeReport.testDatanodeReport(TestDatanodeReport.java:144)
> {noformat}
> The test shuts down datanode, and then shutdown cluster, which shuts down the 
> a datanode twice. Resetting the FsDatasetSpi reference in DataNode to null 
> resolves the issue.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16428) Source path with storagePolicy cause wrong typeConsumed while rename

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16428.
---
Fix Version/s: 3.3.3
   (was: 3.4.0)
   (was: 3.3.4)
   Resolution: Fixed

FIxed in 3.3.3; updating fix versions as appropriate

> Source path with storagePolicy cause wrong typeConsumed while rename
> 
>
> Key: HDFS-16428
> URL: https://issues.apache.org/jira/browse/HDFS-16428
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs, namenode
>Reporter: lei w
>Assignee: lei w
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.3.3, 3.2.3
>
> Attachments: example.txt
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> When compute quota in rename operation , we use storage policy of the target 
> directory to compute src  quota usage. This will cause wrong value of 
> typeConsumed when source path was setted storage policy. I provided a unit 
> test to present this situation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16507.
---
Fix Version/s: 3.3.3
   Resolution: Fixed

FIxed in 3.3.3; updating fix versions as appropriate

> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL 
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to 
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN 
> rolls edit its self. 
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
>     org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>     
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>     
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>     
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
>     java.security.AccessController.doPrivileged(Native Method)
>     javax.security.auth.Subject.doAs(Subject.java:422)
>     
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>     org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>     
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>     
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>     
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>     org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>     
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>     
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>     
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>     org.eclipse.jetty.server.Server.handle(Server.java:539)
>     org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
>     
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>     
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
>     org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
>     
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>     
> org.eclipse.jetty.util.thread

[jira] [Resolved] (HDFS-16422) Fix thread safety of EC decoding during concurrent preads

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16422.
---
Fix Version/s: 3.3.3
   (was: 3.4.0)
   (was: 3.3.4)
   Resolution: Fixed

in release 3.3.3

note that in branch-3.3+ the commit message doesn't include the JIRA ID. added 
during the backport

> Fix thread safety of EC decoding during concurrent preads
> -
>
> Key: HDFS-16422
> URL: https://issues.apache.org/jira/browse/HDFS-16422
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: dfsclient, ec, erasure-coding
>Affects Versions: 3.3.0, 3.3.1
>Reporter: daimin
>Assignee: daimin
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 3.3.3, 3.2.3
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Reading data on an erasure-coded file with missing replicas(internal block of 
> block group) will cause online reconstruction: read dataUnits part of data 
> and decode them into the target missing data. Each DFSStripedInputStream 
> object has a RawErasureDecoder object, and when we doing pread concurrently, 
> RawErasureDecoder.decode will be invoked concurrently too. 
> RawErasureDecoder.decode is not thread safe, as a result of that we get wrong 
> data from pread occasionally.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[ANNOUNCE] branch-3.3.3 created off 3.3.2; branch-3.3 hadoop.version is now 3.3.4-SNAPSHOT

[Steve Loughran]

There is now a branch-3.3.3 in the repo; this is forked off 3.3.2, *not*
branch-3.3

branch-3.3 has its hadoop version now set to 3.3.4

I've renamed the release version 3.3.3 in JIRA to 3.3.4, so all
fixed/target/affects references have been automatically updated. text
references in JIRAs and PRs will not. If you have JIRAs with 3.3. in the
title, now is the time to change them.

There is now a new 3.3.3 version in JIRA for all fixes only targeted at
this release, and for future bug reports.

The list of JIRAs to go in is under
https://issues.apache.org/jira/browse/HADOOP-18198 ; I've re-opened them
all to track the merge

-Steve

[jira] [Reopened] (HDFS-16355) Improve the description of dfs.block.scanner.volume.bytes.per.second

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HDFS-16355:
---

> Improve the description of dfs.block.scanner.volume.bytes.per.second
> 
>
> Key: HDFS-16355
> URL: https://issues.apache.org/jira/browse/HDFS-16355
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, hdfs
>Affects Versions: 3.3.1
>Reporter: guophilipse
>Assignee: guophilipse
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> datanode block scanner will be disabled if 
> `dfs.block.scanner.volume.bytes.per.second` is configured less then or equal 
> to zero, we can improve the desciption



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-16501) Print the exception when reporting a bad block

[Steve Loughran (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reopened HDFS-16501:
---

> Print the exception when reporting a bad block
> --
>
> Key: HDFS-16501
> URL: https://issues.apache.org/jira/browse/HDFS-16501
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Attachments: image-2022-03-10-19-27-31-622.png
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> !image-2022-03-10-19-27-31-622.png|width=847,height=27!
> Currently, volumeScanner will find bad block and report it to namenode 
> without printing the reason why the block is a bad block. I think we should 
> be better print the exception in log file.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org