Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Ayush Saxena]

+1 for the proposal, the first thing might be to just drop the
unnecessary branches which are either from dependabot or someone
accidentally created a branch in the main repo rather than in their
fork, there are many, I don't think we need them in the archived repo
either

Regarding (3) If you mean just branches, then should be ok, maybe lets
not touch the release tags for now IMO

Regrading the regex, I tried this locally & it works
```
ayushsaxena@ayushsaxena hadoop % git tag --delete  `git tag --list 'ozone*'`
Deleted tag 'ozone-0.2.1-alpha-RC0' (was 90b070452bc7)
Deleted tag 'ozone-0.3.0-alpha' (was e9921ebf7e8d)
Deleted tag 'ozone-0.3.0-alpha-RC0' (was 3fbd1f15b894)
Deleted tag 'ozone-0.3.0-alpha-RC1' (was cdad29240e52)
Deleted tag 'ozone-0.4.0-alpha-RC0' (was 07fd26ef6d8c)
Deleted tag 'ozone-0.4.0-alpha-RC1' (was c4f9a20bbe55)
Deleted tag 'ozone-0.4.0-alpha-RC2' (was 6860c595ed19)
Deleted tag 'ozone-0.4.1-alpha' (was 687173ff4be4)
Deleted tag 'ozone-0.4.1-alpha-RC0' (was 9062dac447c8)
ayushsaxena@ayushsaxena hadoop %

```

-Ayush

On Wed, 29 May 2024 at 18:07, Steve Loughran
 wrote:
>
> I'm waiting for a git update of trunk to complete, not having done it since
> last week. The 1.8 GB download is taking a long time over a VPN.
>
> Updating files: 100% (8518/8518), done.
> Switched to branch 'trunk'
> Your branch is up to date with 'apache/trunk'.
> remote: Enumerating objects: 4142992, done.
> remote: Counting objects: 100% (4142972/4142972), done.
> remote: Compressing objects: 100% (503038/503038), done.
> ^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
> remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
> Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
> Resolving deltas:  42% (1505182/3583765)
> ...
>
>
> We have too many branches and too many tags, which makes for big downloads
> and slow clones, as well as complaints from git whenever I manually push
> things to gitbox.apache.org.
>
> I think we can/should clean up, which can be done as
>
>
>1. Create a hadoop-source-archive repository,
>2. into which we add all of the current hadoop-repository. This ensures
>all the history is preserved.
>3. Delete all the old release branches, where old is defined as, maybe <
>2.6?
>4. feature branches which are merged/abandoned
>5. all the single JIRA branches which are the same thing, "MR-279" being
>a key example. ozone-* probably too.
>6. Do some tag pruning too. (Is there a way to do this with wildcards? I
>could use it locally...)
>
> With an archive repo, all the old development history for branches off the
> current release chain + tags are still available, but the core repo is
> much, much smaller.
>
> What do people think?
>
> If others are interested, I'll need some help carefully getting the
> hadoop-source-archive repo up. We'd need to somehow get all of hadoop trunk
> into it.
>
> Meanwhile, I will cull some merged feature branches.
>
> Steve

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Hadoop Committer - Haiyang Hu

[Ayush Saxena]

Congratulations Haiyang!!!

-Ayush

> On 22 Apr 2024, at 9:41 AM, Xiaoqiao He  wrote:
> 
> I am pleased to announce that Haiyang Hu has been elected as
> a committer on the Apache Hadoop project. We appreciate all of
> Haiyang's work, and look forward to her/his continued contributions.
> 
> Congratulations and Welcome, Haiyang!
> 
> Best Regards,
> - He Xiaoqiao
> (On behalf of the Apache Hadoop PMC)
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-11671) Tests in hadoop-yarn-server-router are not running

[Ayush Saxena (Jira)]

Ayush Saxena created YARN-11671:
---

 Summary: Tests in hadoop-yarn-server-router are not running
 Key: YARN-11671
 URL: https://issues.apache.org/jira/browse/YARN-11671
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Ayush Saxena


[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1549/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt]

 
{noformat}
[INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ 
hadoop-yarn-server-router ---
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

>  Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.

Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:

"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."

So, do put an explicit +1, before you count yourself. Good Luck!!!

-Ayush

[1] https://www.apache.org/foundation/voting.html#ReleaseVotes

On Tue, 12 Mar 2024 at 17:27, Steve Loughran 
wrote:

> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop:  true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib:true /lib/aarch64-linux-gnu/libz.so.1
> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L:   false libhadoop was built without ISA-L support
> PMDK:false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release" step, validating maven
> > > artifacts
> > > > * building the same downstream modules which built for me last time
> > (avro
> > > > too complex; hboss not aws v2 in apache yet)
> > > >
> > > > spark build is still ongoing, but I'm not going to wait. It is
> > building,
> > > > which is key.
> > > >
> > > > The core changes I needed in are at the dependency level and I've
> > > > verified they are good.
> > > >
> > > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > > stuff for its checknative; not expecting problems.
> > > >
> > > > So: i've got some stuff still 

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Ayush Saxena]

Thanx Everyone. I think we have an agreement around dropping the Hbase v1
support.

I have created a ticket: https://issues.apache.org/jira/browse/HADOOP-19107

FYI. The HBase v2 build is broken now. I tried " mvn clean install
-DskipTests -Dhbase.profile=2.0" & it didn't work (It worked couple of
months back IIRC), Some sl4j exclusion needs to be added I believe

Maybe an upgrade of the HBase v2 version might solve it, if not we need to
handle it in our code. We can chase the upgrade of v2 version as Bryan
mentioned either in a different ticket parallely or together with this as
well :-)

In case anyone has objections/suggestions do shout out here or on the
ticket or both

-Ayush

On Mon, 11 Mar 2024 at 19:22, Steve Loughran 
wrote:

>  +1 for cutting hbase 1; it only reduces dependency pain (no more protobuf
> 2.5!)
>
> Created the JIRA on that a few days back
> https://issues.apache.org/jira/browse/YARN-11658
>
> On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault 
> wrote:
>
> > Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
> > you are at it you should probably update the hbase2 version, because
> 2.2.x
> > is also very old and EOL. 2.5.x is the currently maintained release for
> > hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
> > well.
> >
> > On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena  wrote:
> >
> > > Hi Folks,
> > > As of now we have two profiles for HBase: one for HBase v1(1.7.1) &
> other
> > > for v2(2.2.4). The versions are specified over here: [1], how to build
> is
> > > mentioned over here: [2]
> > >
> > > As of now we by default run our Jenkins "only" for HBase v1, so we have
> > > seen HBase v2 profile silently breaking a couple of times.
> > >
> > > Considering there are stable versions for HBase v2 as per [3] & HBase
> v2
> > > seems not too new, I have some suggestions, we can consider:
> > >
> > > * Make HBase v2 profile as the default profile & let HBase v1 profile
> > stay
> > > in our code.
> > > * Ditch HBase v1 profile & just lets support HBase v2 profile.
> > > * Let everything stay as is, just add a Jenkins job/ Github action
> which
> > > compiles HBase v2 as well, so we make sure no change breaks it.
> > >
> > > Personally I would go with the second option, the last HBase v1 release
> > > seems to be 2 years back, it might be pulling in some
> > > problematic transitive dependencies & it will open scope for us to
> > support
> > > HBase 3.x when they have a stable release in future.
> > >
> > >
> > > Let me know your thoughts!!!
> > >
> > > -Ayush
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
> > >
> > > [2]
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
> > >
> > > [3] https://hbase.apache.org/downloads.html
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist

I think it due to HADOOP-17524
<https://issues.apache.org/jira/browse/HADOOP-17524>, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon

I will abstain

-Ayush

On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He  wrote:

> cc @Chao Sun  @zhang...@apache.org
>  @Ayush Saxena 
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He  wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:
>>
>>> @Xiaoqiao He  @Ayush Saxena 
>>> @Steve
>>> Loughran  @Mukund Madhav Thakur
>>>  @Takanobu
>>> Asanuma  @Masatake Iwasaki 
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>

[DISCUSS] Support/Fate of HBase v1 in Hadoop

[Ayush Saxena]

Hi Folks,
As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
for v2(2.2.4). The versions are specified over here: [1], how to build is
mentioned over here: [2]

As of now we by default run our Jenkins "only" for HBase v1, so we have
seen HBase v2 profile silently breaking a couple of times.

Considering there are stable versions for HBase v2 as per [3] & HBase v2
seems not too new, I have some suggestions, we can consider:

* Make HBase v2 profile as the default profile & let HBase v1 profile stay
in our code.
* Ditch HBase v1 profile & just lets support HBase v2 profile.
* Let everything stay as is, just add a Jenkins job/ Github action which
compiles HBase v2 as well, so we make sure no change breaks it.

Personally I would go with the second option, the last HBase v1 release
seems to be 2 years back, it might be pulling in some
problematic transitive dependencies & it will open scope for us to support
HBase 3.x when they have a stable release in future.


Let me know your thoughts!!!

-Ayush


[1]
https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207

[2]
https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172

[3] https://hbase.apache.org/downloads.html

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[Ayush Saxena]

+1 (Binding)

* Built from source
* Verified Checksums
* Verified Signatures
* Verified no diff b/w the git tag & src tar
* Verified LICENSE & NOTICE files
* Compiled hadoop trunk with 1.2.0-RC1*

Thanx Shilun Fan & Xiaoqiao He for driving the release. Good Luck!!!

* You need to change that protobuf artifact name to compile, that ain't
fancy at all, the exclusions and all need to be updated after every
release, this thing we should change maybe from next release and let the
dependency stay without the version suffix. Incompatible for sure, but the
current approach isn't very compatible either, If someone is having a
dependency of any hadoop module, which pulls in this shaded jar
transitively, that guy needs to update his exclusions post every release
and that IMO isn't a very good experience, lets chase that in our next
release or have some discussions around it

-Ayush


On Tue, 6 Feb 2024 at 19:14, Takanobu Asanuma  wrote:

> +1 (binding).
>
> * Verified signatures and checksums
> * Reviewed the documents
> * Successfully built from source with `mvn clean install`
> * Successfully compiled Hadoop trunk and branch-3.4 with `mvn clean install
> -DskipTests` using the thirdparty 1.2.0-RC1
> * There are not any diffs between tag and src
>
> Regards,
> - Takanobu Asanuma
>
> 2024年2月6日(火) 11:03 Xiaoqiao He :
>
> > cc @PJ Fanning  @Ayush Saxena 
> @Steve
> > Loughran  @Takanobu Asanuma 
> @Shuyan
> > Zhang  @inigo...@apache.org  >
> >
> > On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He 
> wrote:
> >
> >> +1(binding).
> >>
> >> I checked the following items:
> >> - [X] Download links are valid.
> >> - [X] Checksums and PGP signatures are valid.
> >> - [X] LICENSE and NOTICE files are correct for the repository.
> >> - [X] Source code artifacts have correct names matching the current
> >> release.
> >> - [X] All files have license headers if necessary.
> >> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
> >> - [X] Built Hadoop trunk successfully with updated thirdparty (include
> >> update protobuf shaded path).
> >> - [X] No difference between tag and release src tar.
> >>
> >> Good Luck!
> >>
> >> Best Regards,
> >> - He Xiaoqiao
> >>
> >>
> >> On Sun, Feb 4, 2024 at 10:29 PM slfan1989  wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Xiaoqiao He and I have put together a release candidate (RC1) for
> Hadoop
> >>> Thirdparty 1.2.0.
> >>>
> >>> The RC is available at:
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
> >>>
> >>> The RC tag is
> >>>
> >>>
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
> >>>
> >>> The maven artifacts are staged at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1401
> >>>
> >>> Comparing to 1.1.1, there are three additional fixes:
> >>>
> >>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> >>> https://github.com/apache/hadoop-thirdparty/pull/26
> >>>
> >>> HADOOP-18921. Upgrade to avro 1.11.3
> >>> https://github.com/apache/hadoop-thirdparty/pull/24
> >>>
> >>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
> >>> https://github.com/apache/hadoop-thirdparty/pull/23
> >>>
> >>> You can find my public key at :
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>
> >>> Best Regards,
> >>> Shilun Fan.
> >>>
> >>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Ayush Saxena]

There is some diff b/w the git tag & the src tar, the Dockerfile & the
create-release are different, Why?

Files hadoop-thirdparty/dev-support/bin/create-release and
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ

Files hadoop-thirdparty/dev-support/docker/Dockerfile and
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ


ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/bin/create-release
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release

444,446c444,446

< echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"

< echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"

< echo "RUN chown -R ${user_name} /home/${user_name}"

---

> echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit 0;"

> echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}; exit
0;"

> echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"

ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/docker/Dockerfile
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile

103a104,105

> RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
/etc/maven/settings.xml

>

126a129,130

> RUN pip2 install setuptools-scm==5.0.2

> RUN pip2 install lazy-object-proxy==1.5.0

159d162

<




Other things look Ok,
* Built from source
* Verified Checksums
* Verified Signatures
* Validated files have ASF header

Not sure if having diff b/w the git tag & src tar is ok, this doesn't look
like core code change though, can anybody check & confirm?

-Ayush


On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:

> Gentle ping. @Ayush Saxena  @Steve Loughran
>  @inigo...@apache.org  @Masatake
> Iwasaki  and some other folks.
>
> On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:
>
> > Thank you for the review and vote! Looking forward to other forks helping
> > with voting and verification.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> wrote:
> >
> > > Thanks Shilun for driving it and making it happen.
> > >
> > > +1(binding).
> > >
> > > [x] Checksums and PGP signatures are valid.
> > > [x] LICENSE files exist.
> > > [x] NOTICE is included.
> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > > [x] Built from source works well: `mvn clean install`
> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
> > update
> > > protobuf shaded path).
> > >
> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> > > could finish this vote before 2024/02/06(UTC) if there are no concerns.
> > > Thanks all.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >
> > >
> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
> > Hadoop
> > > > Thirdparty 1.2.0.
> > > >
> > > > The RC is available at:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > > >
> > > > The RC tag is
> > > >
> > >
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
> > > >
> > > > Comparing to 1.1.1, there are three additional fixes:
> > > >
> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > > > https://github.com/apache/hadoop-thirdparty/pull/26
> > > >
> > > > HADOOP-18921. Upgrade to avro 1.11.3
> > > > https://github.com/apache/hadoop-thirdparty/pull/24
> > > >
> > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > > > https://github.com/apache/hadoop-thirdparty/pull/23
> > > >
> > > > You can find my public key at :
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Ayush Saxena]

We should consider including
https://issues.apache.org/jira/browse/HDFS-17129

Which looks like inducing some misorder between IBR & FBR which can
potentially lead to strange issues, if that can’t be merged, should revert
the one which causes that.

I think we should check for any ticket which has a target version or
affect version & is marked critica/blockerl for 3.4.0 before spinning up a
new RC, I think I mentioned that somewhere before.

-1, in case HDFS-17129 is not a false alarm or we can prove it won't cause
any issues. There is a comment which says a block was reported missing post
the patch that induced it: [1]

[1] https://github.com/apache/hadoop/pull/6244#issuecomment-1793981740

-Ayush


On Fri, 12 Jan 2024 at 07:37, slfan1989  wrote:

> Thank you very much for your help in verifying this version! We will use
> version 3.5.0 for fix jira in the future.
>
> Best Regards,
> Shilun Fan.
>
>  > wonderful! I'll be testing over the weekend
>
>  > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in
> 3.5.0
>  > -correct?
>
>  > steve
>
>
> > On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote:
>
> > Hello all,
> >
> > We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
> > hadoop 3.4.0-RC version.
> >
> > The RC is available at:
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
> >
> > Maven artifacts is built by x86 machine and are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1391/
> >
> > My public key:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Changelog:
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
> >
> > Release notes:
> >
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
> >
> > This is a relatively big release (by Hadoop standard) containing about
> 2852
> > commits.
> >
> > Please give it a try, this RC vote will run for 7 days.
> >
> > Feature highlights:
> >
> > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
> > 
> > [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
> > FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
> > FsDatasetImpl datasetLock
> > When there are many namespaces in a large cluster.
> >
> > YARN Federation improvements
> > 
> > [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
> > improvements, including the following:
> >
> > 1. YARN Router now boasts a full implementation of all relevant
> interfaces
> > including the ApplicationClientProtocol,
> > ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
> > 2. Enhanced support for Application cleanup and automatic offline
> > mechanisms for SubCluster are now facilitated by the YARN Router.
> > 3. Code optimization for Router and AMRMProxy was undertaken, coupled
> with
> > improvements to previously pending functionalities.
> > 4. Audit logs and Metrics for Router received upgrades.
> > 5. A boost in cluster security features was achieved, with the inclusion
> of
> > Kerberos support.
> > 6. The page function of the router has been enhanced.
> >
> > Upgrade AWS SDK to V2
> > 
> > [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
> > The S3A connector now uses the V2 AWS SDK. This is a significant change
> at
> > the source code level.
> > Any applications using the internal extension/override points in the
> > filesystem connector are likely to break.
> > Consult the document aws\_sdk\_upgrade for the full details.
> >
> > hadoop-thirdparty will also provide the new RC0 soon.
> >
> > Best Regards,
> > Shilun Fan.
> >
>

Re: Re: [DISCUSS] Release Hadoop 3.4.0

[Ayush Saxena]

Thanx @slfan1989 for volunteering. Please remove this [1] from the new
branches-3.4 & 3.4.0 when you create them as part of preparing for the
release, else it would be putting up trunk labels for backport PRs to
those branches as well.

There are some tickets marked as Critical/Blocker for 3.4.0 [2], just
give a check to them if they are actually critical or not, if yes, we
should get them in. Most of them were not looking relevant to me at
first glance.

-Ayush


[1] https://github.com/apache/hadoop/blob/trunk/.github/labeler.yml
[2] 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20MAPREDUCE%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(3.4.0%2C%203.4.1)


On Thu, 4 Jan 2024 at 19:57, slfan1989  wrote:
>
> Hey all,
>
> We are planning to release Hadoop 3.4.0 base on trunk. I made some 
> preparations and changed the target version of JIRA for non-blockers in 
> HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create a 
> new JIRA, the target version can directly select version 3.5.0.
>
> If you have any thoughts, suggestions, or concerns, please feel free to share 
> them.
>
> Best Regards,
> Shilun Fan.
>
> > +1 from me.
> >> It will include the new AWS V2 SDK upgrade as well.
>
> > On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote:
>
> > >
> > > I think the release discussion can be in public ML?
> >
> > Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote:
> >
> > > +1 from me as well.
> > >
> > > We should definitely attempt to upgrade the thirdparty version for
> > > 3.4.0 & check if there are any pending critical/blocker issues as
> > > well.
> > >
> > > I think the release discussion can be in public ML?
> > >
> > > -Ayush
> > >
> > > On Mon, 1 Jan 2024 at 18:25, Steve Loughran  > >
> > > wrote:
> > > >
> > > > +1 from me
> > > >
> > > > ant and maven repo to build and validate things, including making arm
> > > > binaries if you work from an arm macbook.
> > > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > > >
> > > > do we need to publish an up to date thirdparty release for this?
> > > >
> > > >
> > > >
> > > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote:
> > > >
> > > > > Dear PMC Members,
> > > > >
> > > > > First of all, Merry Christmas to everyone!
> > > > >
> > > > > In our community discussions, we collectively finalized the plan to
> > > release
> > > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to take
> > > on
> > > > > the responsibility for the initial release of version 3.4.0, and the
> > > entire
> > > > > process is set to officially commence in January 2024.
> > > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > > >
> > > > > The specific work plan includes:
> > > > >
> > > > > 1. Following the guidance in the HowToRelease document, completing
> > all
> > > the
> > > > > relevant tasks required for the release of version 3.4.0.
> > > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > > > 3. Currently, the Fix Versions of all tasks merged into trunk are set
> > > as
> > > > > 3.4.0; I will move them to 3.5.0.
> > > > >
> > > > > Confirmed features to be included in the release:
> > > > >
> > > > > 1. Enhanced functionality for YARN Federation.
> > > > > 2. Optimization of HDFS RBF.
> > > > > 3. Introduction of fine-grained global locks for DataNodes.
> > > > > 4. Improvements in the stability of HDFS EC, and more.
> > > > > 5. Fixes for important CVEs.
> > > > >
> > > > > If you have any thoughts, suggestions, or concerns, please feel free
> > to
> > > > > share them.
> > > > >
> > > > > Looking forward to a successful release!
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > >
> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Hadoop 3.2.x EOL

[Ayush Saxena]

+1

-Ayush

> On 06-Dec-2023, at 9:40 AM, Xiaoqiao He  wrote:
> 
> Dear Hadoop devs,
> 
> Given the feedback from the discussion thread [1], I'd like to start
> an official thread for the community to vote on release line 3.2 EOL.
> 
> It will include,
> a. An official announcement informs no further regular Hadoop 3.2.x
> releases.
> b. Issues which target 3.2.5 will not be fixed.
> 
> This vote will run for 7 days and conclude by Dec 13, 2023.
> 
> I’ll start with my +1.
> 
> Best Regards,
> - He Xiaoqiao
> 
> [1] https://lists.apache.org/thread/bbf546c6jz0og3xcl9l3qfjo93b65szr

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Make some release lines EOL

[Ayush Saxena]

Thanx He Xiaoqiao for starting the thread.

+1 to mark 3.2 EOL. I am not sure about branch-2 stuff, I think a
bunch of folks are still using Hadoop 2.x, but we hardly push anything
over there & I am not sure if anyone is interested in releasing that
or not.

> Hadoop 3.3 Release (release-3.3.5 at Jun 22 2022),  360 commits checked in
since last release.

I think you got the year wrong here, it is 2023

-Ayush

On Mon, 4 Dec 2023 at 18:09, Xiaoqiao He  wrote:
>
> Hi folks,
>
> There are many discussions about which release lines should we still
> consider actively
> maintained in history. I want to launch this topic again, and try to get a
> consensus.
>
> From download page[1] and active branches page[2], we have the following
> release lines:
> Hadoop 3.3 Release (release-3.3.5 at Jun 22 2022),  360 commits checked in
> since last release.
> Hadoop 3.2 Release (release-3.2.4 at Jul 11, 2022) 36 commits checked in
> since last release.
> Hadoop 2.10 Release (release-2.10.2 at May 17, 2022) 24 commits checked in
> since last release.
>
> And Hadoop 3.4.0 will be coming soon which Shilun Fan (maybe cooperating
> with Ahmar Suhail?)
> has been actively working on getting the 3.4.0 release out.
>
> Considering the less updates for some active branches, should we declare to
> our downstream
> users that some of these lines will EOL?
>
> IMO we should announce EOL branch-2.10 and branch-3.2 which are not active
> now.
> Then we could focus on minor active branches (branch-3.3 and branch-3.4)
> and increase release pace.
>
> So how about to keep branch-3.3 and branch-3.4 release lines as actively
> maintained, And mark branch-2.10 and branch-3.2 EOL? Any opinions? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://hadoop.apache.org/releases.html
> [2]
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.4.0 release

[Ayush Saxena]

Hi Ahmar,

Makes sense to have a 3.4.0 rather than a 3.3.x if there are major
changes. Few things on the process:

* trunk should now point to 3.5.0-SNAPSHOT
* Fix Versions of all the tickets merged to trunk would be 3.4.0 as of
now, you need to move it to 3.5.0, else while chasing the release, the
release notes will get messed up.
* Since we are moving to 3.4.x, we can even explore a few more changes
in trunk as well, which weren't merged to 3.3.x due to compat issues.
HADOOP-13386 is one such change, which was asked for some time back.
* Now we have an additional release line, we can start a thread and
think about marking some existing release lines as EOL, so, the number
of active release lines stay in control, maybe 3.2

Rest, Good Luck, Thanx for volunteering!!!

-Ayush

On Fri, 1 Dec 2023 at 17:30, Ahmar Suhail  wrote:
>
> Hey all,
>
>
> HADOOP-18073 was recently merged to trunk, which upgrades the AWS SDK to
> V2. Since this merge, there has been work for stabilising this SDK version
> upgrade, as tracked in HADOOP-18886. Further, HADOOP-18995 and HADOOP-18996
> added support for Amazon Express One Zone.
>
>
> The SDK upgrade has brought major code changes to the hadoop-aws module.
> For example, In the v2 SDK, all package, class and method names have
> changed. Due to the upgrade being a large change, it might be good to
> release these changes as Hadoop 3.4.0. For this, I started creating a
> branch-3.4 from branch-3.3 and branch-3.4.0 for a possible release early
> next year.
>
>
> The other option would be to release from trunk as a 3.3.x release, but due
> to the large number of changes there this will be really difficult.
>
>
> Thoughts on a 3.4.0 release?

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-11607) TestTimelineAuthFilterForV2 fails intermittently

[Ayush Saxena (Jira)]

Ayush Saxena created YARN-11607:
---

 Summary: TestTimelineAuthFilterForV2 fails intermittently 
 Key: YARN-11607
 URL: https://issues.apache.org/jira/browse/YARN-11607
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Ayush Saxena


Ref:
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1398/testReport/junit/org.apache.hadoop.yarn.server.timelineservice.security/TestTimelineAuthFilterForV2/testPutTimelineEntities_boolean__boolean__3_/

{noformat}
org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
at 
org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:527)
at 
org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.publishAndVerifyEntity(TestTimelineAuthFilterForV2.java:324)
at 
org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.publishWithRetries(TestTimelineAuthFilterForV2.java:337)
at 
org.apache.hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2.testPutTimelineEntities(TestTimelineAuthFilterForV2.java:383)
{noformat}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Hadoop PMC Member - Shilun Fan

[Ayush Saxena]

Congratulations!!!

-Ayush

> On 31-Oct-2023, at 6:15 PM, Samrat Deb  wrote:
> 
> Congratulations Shilun Fan
> 
> Bests,
> Samrat
> 
>> On Tue, Oct 31, 2023 at 6:07 PM Tao Li  wrote:
>> 
>> Congratulations!!!
>> 
>> Xiaoqiao He  于2023年10月31日周二 20:19写道：
>> 
>>> On behalf of the Apache Hadoop PMC, I am pleased to announce that
>>> Shilun Fan(slfan1989) has accepted the PMC's invitation to become
>>> PMC member on the project. We appreciate all of Shilun's generous
>>> contributions thus far and look forward to his continued involvement.
>>> 
>>> Congratulations and welcome, Shilun!
>>> 
>>> Best Regards,
>>> He Xiaoqiao
>>> (On behalf of the Apache Hadoop PMC)
>>> 
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE

[Ayush Saxena]

POM change only & it has no test failing, so good from my side(HDFS)

-Ayusg

> On 16-Oct-2023, at 4:48 PM, Steve Loughran  
> wrote:
> 
> Can anyone from yarn or hdfs review this before i just upvote it hoping all
> will be well. thanks
> 
> -- Forwarded message -
> From: PJ Fanning (Jira) 
> Date: Wed, 11 Oct 2023 at 20:27
> Subject: [jira] [Created] (HADOOP-18933) upgrade netty to 4.1.100 due to CVE
> To: 
> 
> 
> PJ Fanning created HADOOP-18933:
> ---
> 
> Summary: upgrade netty to 4.1.100 due to CVE
> Key: HADOOP-18933
> URL: https://issues.apache.org/jira/browse/HADOOP-18933
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: PJ Fanning
> 
> 
> follow up to https://issues.apache.org/jira/browse/HADOOP-18783
> 
> https://netty.io/news/2023/10/10/4-1-100-Final.html
> 
> 
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.20.10#820010)
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Hadoop Committer - Simbarashe Dzinamarira

[Ayush Saxena]

Congratulations!!!

-Ayush

> On 03-Oct-2023, at 5:42 AM, Erik Krogen  wrote:
> 
> Congratulations Simba! Thanks for the great work you've done on making HDFS
> more scalable!
> 
>> On Mon, Oct 2, 2023 at 4:31 PM Iñigo Goiri  wrote:
>> 
>> I am pleased to announce that Simbarashe Dzinamarira has been elected as a
>> committer on the Apache Hadoop project.
>> We appreciate all of Simbarashe's work, and look forward to his continued
>> contributions.
>> 
>> Congratulations and welcome !
>> 
>> Best Regards,
>> Inigo Goiri
>> (On behalf of the Apache Hadoop PMC)
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Hadoop Committer - Shuyan Zhang

[Ayush Saxena]

Congratulations!!!

Sent from my iPhone

> On 27-Sep-2023, at 9:56 AM, Xiaoqiao He  wrote:
> 
> I am pleased to announce that Shuyan Zhang has been elected as
> a committer on the Apache Hadoop project. We appreciate all of
> Shuyan's work, and look forward to her/his continued contributions.
> 
> Congratulations and Welcome, Shuyan!
> 
> Best Regards,
> - He Xiaoqiao
> (On behalf of the Apache Hadoop PMC)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: HADOOP-18207 hadoop-logging module about to land

[Ayush Saxena]

gt; "site configs" to "log4j properties", we don't seem to have any lost
> abilities. This is also not a lost ability, it's the way of configuring it
> that is different.
>
>
>> Most of the time when this entire activity breaks & like usual we are on
> a follow-up or on an addendum PR, there is generally some sarcastic or a
> response like: 'We can't do it without breaking things', and I am not
> taking any of these for now.
>
> My sincere apologies if you think I have not addressed your comments. Could
> you please provide a specific reference and I will be happy to provide
> detailed answers. Many of the questions we are dealing with at this point
> in time have already been discussed on the parent Jira in the past.
>
> I would also like to share some of my opinions: I understand that none of
> this is simpler to deal with, none of this is an interesting work either,
> as opposed to working on a big feature or providing some bug fixes
> (specifically when it takes hours and days to figure out where we have the
> bug, despite how small the fix might be), however we still cannot give up
> on the work that helps maintain our project, can we?
> Just to provide an example, we don't have Java 11 compile support because
> of (let's say) the lack of migration from Jersey 1.x to 2.x (HADOOP-15984
> <https://issues.apache.org/jira/browse/HADOOP-15984>). At this point, it
> seems extremely difficult to migrate to Jersey 2 because of multiple
> factors (e.g. guice support + hk2 dependency injection don't work well in
> Jersey 2), I have initiated a mailing thread on eclipse community and also
> created an issue on jersey tracker to get some insights:
> https://github.com/eclipse-ee4j/jersey/issues/5357
> There has been no update and it is well known that guice support does not
> work with jersey 2, we are not aware of which direction we are going to go
> with, and how can we possibly upgrade jersey, but we still cannot give up
> on it, right? We do need to find some way, don't we?
> Similarly, the whole log4j2 upgrade is also a big deal, but we are not
> going to lose any hadoop functionality, we have an alternative way of
> achieving the same output (as mentioned about async logger).
>
> The point I am trying to make is: all this work is really not that
> interesting, but we still need to work on it, we still need to
> maintain hadoop project for the rest of the world (no matter how complex it
> is to maintain it) as a community, correct? Hadoop is the only project that
> doesn't have log4j2 support among all the other big data projects that we
> are using as of today, we can stay in this state for a while but how long
> are we willing to compromise an inevitable boring maintenance effort?
> Moreover, once we migrate to log4j2, we could get a real boost from some of
> their async logger efforts, where we don't even need async appender.
>
> I would be extremely happy if anyone else is willing to put efforts and get
> this work rolling forward. I am committed to ensuring that my work on the
> project does not give rise to "multiple community conflicts." Therefore, if
> we can confidently determine that using log4j1 for the foreseeable future
> is acceptable, there would be no need for additional dev and review cycles.
> I am more than willing to refrain from creating any further patches in such
> a scenario. Similar discussion is required for jersey as well, while it
> might be tempting to hope that integrating jersey 2 will magically resolve
> all our issues and make our Jackson and other dependency upgrades reliable,
> it is essential to be realistic about the potential challenges ahead.
> Despite the appeal of adopting jersey 2, we must be prepared to face a
> substantial num of incompatibilities that would arise with jersey 2
> migration.
>
> I am really grateful to all reviewers that have reviewed the tasks so far,
> I am not expecting reviewers to be able to provide their reviews as quickly
> as possible, this particular sub-task has been dev and test ready for the
> last 4 months, and it is absolutely okay to wait longer, but what really
> hurts a bit is the fact that despite the whole discussion that took place
> on the parent Jira, and the clear agreements/directions we have agreed
> upon, we are still engaging in the discussion to determine the value this
> work brings in.
> I sincerely apologize if any aspects were not adequately clarified during
> our discussions of each sub-task. I am more than willing to revisit any
> line of code and engage in a detailed conversation to share insights into
> the factors that influenced the changes made.
>
>
> 1. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 2. https://lists.apache

Re: HADOOP-18207 hadoop-logging module about to land

[Ayush Saxena]

Hi Wei-Chiu,
I am glad this activity finally made it to the dev mailing list. Just
sharing the context being the guy who actually reverted this last time
it was in: It had a test failure on the PR itself and it went in, that
had nothing to do with the nature of the PR, generic for all PR and
all projects.

Some thoughts & Questions?
* Regarding this entire activity including the parent tickets: Do we
have any dev list agreement for this?
* What incompatibilities have been introduced till now for this and
what are planned.
* What does this activity bring up for the downstream folks adapting
this? Upgrading Hadoop is indeed important for a lot of projects and
for "us as well" and it is already a big pain (my past experience)
* What tests have been executed verifying all these changes including
this and the ones already in, apart from the Jenkins results, and
what's the plan.
* Considering you are heavily involved, any insights around perf stuff?
* This Comment 
[https://github.com/apache/hadoop/pull/5503#discussion_r1199614640],
this says it isn't moving all the instances? So, when do you plan to
work on this? Should that be a release blocker for us, since part of
the activity is in? Needless to say: "Best Effort, whatever could move
in, moves is, isn't an answer"
* The above comment thread even says losing some available abilities,
even some past one said so, what all is getting compromised, and how
do you plan to get it back? Most of the lost abilities are related to
HDFS, I don't think we are in a state to lose stuff there, if we
aren't having enough to make people adapt. Our ultimate goal isn't to
have something in, but to make people use it.
* What advantages do we get with all of these activities over existing
branch-3 stuff? Considering what are the trade-offs, Was discussing
with some folks offline & that seems to be a good question to have an
answer beforehand.

PS. Most of the time when this entire activity breaks & like usual we
are on a follow-up or on an addendum PR, there is generally some
sarcastic or a response like: 'We can't do it without breaking
things', and I am not taking any of these for now.

Most importantly since we are discussing it now and if there are
incompatibilities introduced already, is there a possible way out and
get rid of them, if not, if there ain't an agreement, how tough is
going back, because if it introduces incompatibilities for HDFS, you
won't get an agreement most probably, not sure about others but I will
veto that...


TLDR, Please hold unless all the concerns are addressed and we have an
agreement for this as well as anything done in past or planned for
future, Shouldn't compromise the adaptability of the product at any
cost

-Ayush

On Thu, 27 Jul 2023 at 03:47, Wei-Chiu Chuang  wrote:
>
> Hi,
>
> I am preparing to resolve HADOOP-18207
>  (
> https://github.com/apache/hadoop/pull/5717).
>
> This change affects all modules. With this change, it will eliminate almost
> all the direct log4j usage.
>
> As always, landing such a big piece is tricky. I am sorry for the mishaps
> last time and am doing more due diligence to make it a smoother transition.
> I am triggering one last precommit check. Once the change is merged, Viraj
> and I will pay attention to any potential problems.
>
> Weichiu

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Signing releases using automated release infra

[Ayush Saxena]

Yep, thirdparty could be a good candidate to try, building thirdparty
release is relatively easy as well

-Ayush

On Thu, 20 Jul 2023 at 15:25, Steve Loughran  wrote:
>
>
> could be good.
>
> why not set it up for the third-party module first to see how well it works?
>
> On Tue, 18 Jul 2023 at 21:05, Ayush Saxena  wrote:
>>
>> Something we can explore as well!!
>>
>> -Ayush
>>
>> Begin forwarded message:
>>
>> > From: Volkan Yazıcı 
>> > Date: 19 July 2023 at 1:24:49 AM IST
>> > To: d...@community.apache.org
>> > Subject: Signing releases using automated release infra
>> > Reply-To: d...@community.apache.org
>> >
>> > Abstract: Signing release artifacts using an automated release
>> > infrastructure has been officially approved by LEGAL. This enables
>> > projects to sign artifacts using, say, GitHub Actions.
>> >
>> > I have been trying to overhaul the Log4j release process and make it
>> > as frictionless as possible since last year. As a part of that effort,
>> > I wanted to sign artifacts in CI during deployment and in a
>> > `members@a.o` thread[0] I explained how one can do that securely with
>> > the help of Infra. That was in December 2022. It has been a long,
>> > rough journey, but we succeeded. In this PR[1], Legal has updated the
>> > release policy to reflect that this process is officially allowed.
>> > Further, Infra put together guides[2][3] to assist projects. Logging
>> > Services PMC has already successfully performed 4 Log4j Tools releases
>> > using this approach, see its release process[4] for a demonstration.
>> >
>> > [0] (members only!)
>> > https://lists.apache.org/thread/1o12mkjrhyl45f9pof94pskg55vhs61n
>> > [1] https://github.com/apache/www-site/pull/235
>> > [2] https://infra.apache.org/release-publishing.html#signing
>> > [3] https://infra.apache.org/release-signing.html#automated-release-signing
>> > [4] 
>> > https://github.com/apache/logging-log4j-tools/blob/master/RELEASING.adoc
>> >
>> > # F.A.Q.
>> >
>> > ## Why shall a project be interested in this?
>> >
>> > It greatly simplifies the release process. See Log4j Tools release
>> > process[4], probably the simplest among all Java-based ASF projects.
>> >
>> > ## How can a project get started?
>> >
>> > 1. Make sure your project builds are reproducible (otherwise there is
>> > no way PMC can verify the integrity of CI-produced and -signed
>> > artifacts)
>> > 2. Clone and adapt INFRA-23996 (GPG keys in GitHub secrets)
>> > 3. Clone and adapt INFRA-23974 (Nexus creds. in GitHub secrets for
>> > snapshot deployments)
>> > 4. Clone and adapt INFRA-24051 (Nexus creds. in GitHub secrets for
>> > staging deployments)
>> >
>> > You might also want to check this[5] GitHub Action workflow for 
>> > inspiration.
>> >
>> > [5] 
>> > https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml
>> >
>> > ## Does the "automated release infrastructure" (CI) perform the full 
>> > release?
>> >
>> > No. CI *only* uploads signed artifacts to Nexus. The release manager
>> > (RM) still needs to copy the CI-generated files to SVN, PMC needs to
>> > vote, and, upon consensus, RM needs to "close" the release in Nexus
>> > and so on.
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>> > For additional commands, e-mail: dev-h...@community.apache.org
>> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11540) Fix typo: form -> from

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11540.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix typo: form -> from
> --
>
> Key: YARN-11540
> URL: https://issues.apache.org/jira/browse/YARN-11540
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Seokchan Yoon
>Assignee: Seokchan Yoon
>Priority: Trivial
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> There is a typo in a log message that NodeManager prints:
> > Retrieved credentials form RM for {}: {}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Fwd: Signing releases using automated release infra

[Ayush Saxena]

Something we can explore as well!!

-Ayush

Begin forwarded message:

> From: Volkan Yazıcı 
> Date: 19 July 2023 at 1:24:49 AM IST
> To: d...@community.apache.org
> Subject: Signing releases using automated release infra
> Reply-To: d...@community.apache.org
> 
> Abstract: Signing release artifacts using an automated release
> infrastructure has been officially approved by LEGAL. This enables
> projects to sign artifacts using, say, GitHub Actions.
> 
> I have been trying to overhaul the Log4j release process and make it
> as frictionless as possible since last year. As a part of that effort,
> I wanted to sign artifacts in CI during deployment and in a
> `members@a.o` thread[0] I explained how one can do that securely with
> the help of Infra. That was in December 2022. It has been a long,
> rough journey, but we succeeded. In this PR[1], Legal has updated the
> release policy to reflect that this process is officially allowed.
> Further, Infra put together guides[2][3] to assist projects. Logging
> Services PMC has already successfully performed 4 Log4j Tools releases
> using this approach, see its release process[4] for a demonstration.
> 
> [0] (members only!)
> https://lists.apache.org/thread/1o12mkjrhyl45f9pof94pskg55vhs61n
> [1] https://github.com/apache/www-site/pull/235
> [2] https://infra.apache.org/release-publishing.html#signing
> [3] https://infra.apache.org/release-signing.html#automated-release-signing
> [4] https://github.com/apache/logging-log4j-tools/blob/master/RELEASING.adoc
> 
> # F.A.Q.
> 
> ## Why shall a project be interested in this?
> 
> It greatly simplifies the release process. See Log4j Tools release
> process[4], probably the simplest among all Java-based ASF projects.
> 
> ## How can a project get started?
> 
> 1. Make sure your project builds are reproducible (otherwise there is
> no way PMC can verify the integrity of CI-produced and -signed
> artifacts)
> 2. Clone and adapt INFRA-23996 (GPG keys in GitHub secrets)
> 3. Clone and adapt INFRA-23974 (Nexus creds. in GitHub secrets for
> snapshot deployments)
> 4. Clone and adapt INFRA-24051 (Nexus creds. in GitHub secrets for
> staging deployments)
> 
> You might also want to check this[5] GitHub Action workflow for inspiration.
> 
> [5] 
> https://github.com/apache/logging-log4j-tools/blob/master/.github/workflows/build.yml
> 
> ## Does the "automated release infrastructure" (CI) perform the full release?
> 
> No. CI *only* uploads signed artifacts to Nexus. The release manager
> (RM) still needs to copy the CI-generated files to SVN, PMC needs to
> vote, and, upon consensus, RM needs to "close" the release in Nexus
> and so on.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Ayush Saxena]

+1 (Binding)

* Built from source (x86 & Arm)
* Successful native build on ubuntu 18.04(x86) & ubuntu 20.04(Arm)
* Verified Checksums (x86 & Arm)
* Verified Signatures (x86 & Arm)
* Successful RAT check (x86 & Arm)
* Verified the diff b/w the tag & the source tar
* Built Ozone with 3.3.6, green build after a retrigger due to some OOM
issues [1]
* Built Tez with 3.3.6 green build [2]
* Ran basic HDFS shell commands (Fs
Operations/EC/RBF/StoragePolicy/Snapshots) (x86 & Arm)
* Ran some basic Yarn shell commands.
* Browsed through the UI (NN, DN, RM, NM, JHS) (x86 & Arm)
* Ran some example Jobs (TeraGen, TeraSort, TeraValidate, WordCount,
WordMean, Pi) (x86 & Arm)
* Verified the output of `hadoop version` (x86 & Arm)
* Ran some HDFS unit tests around FsOperations/EC/Observer Read/RBF/SPS
* Skimmed over the contents of site jar
* Skimmed over the staging repo.
* Checked the NOTICE & Licence files.

Thanx Wei-Chiu for driving the release, Good Luck!!!

-Ayush


[1] https://github.com/ayushtkn/hadoop-ozone/actions/runs/5282707769
[2] https://github.com/apache/tez/pull/285#issuecomment-1590962978

On Sat, 24 Jun 2023 at 09:43, Nilotpal Nandi 
wrote:

> +1 (Non-binding).
> Thanks a lot Wei-Chiu for driving it.
>
> Thanks,
> Nilotpal Nandi
>
> On 2023/06/23 21:51:56 Wei-Chiu Chuang wrote:
> > +1 (binding)
> >
> > Note: according to the Hadoop bylaw, release vote is open for 5 days,
> not 7
> > days. So technically the time is almost up.
> > https://hadoop.apache.org/bylaws#Decision+Making
> >
> > If you plan to cast a vote, please do so soon. In the meantime, I'll
> start
> > to prepare to wrap up the release work.
> >
> > On Fri, Jun 23, 2023 at 6:09 AM Xiaoqiao He 
> wrote:
> >
> > > +1(binding)
> > >
> > > * Verified signature and checksum of all source tarballs.
> > > * Built source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > > -DskipTests -Pnative -Pdist -Dtar`.
> > > * Setup pseudo cluster with HDFS and YARN.
> > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result.
> > > * Run example mr applications and check the result - Pi & wordcount.
> > > * Checked the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> > > * Checked git and JIRA using dev-support tools
> > > `git_jira_fix_version_check.py` .
> > >
> > > Thanks WeiChiu for your work.
> > >
> > > NOTE: I believe the build fatal error report from me above is only
> related
> > > to my own environment.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > On Thu, Jun 22, 2023 at 4:17 PM Chen Yi  wrote:
> > >
> > > > Thanks Wei-Chiu for leading this effort !
> > > >
> > > > +1(Binding)
> > > >
> > > >
> > > > + Verified the signature and checksum of all tarballs.
> > > > + Started a web server and viewed documentation site.
> > > > + Built from the source tarball on macOS 12.3 and OpenJDK 8.
> > > > + Launched a pseudo distributed cluster using released binary
> packages,
> > > > done some HDFS dir/file basic opeations.
> > > > + Run grep, pi and wordcount MR tasks on the pseudo cluster.
> > > >
> > > > Bests,
> > > > Sammi Chen
> > > > 
> > > > 发件人: Wei-Chiu Chuang 
> > > > 发送时间: 2023年6月19日 8:52
> > > > 收件人: Hadoop Common ; Hdfs-dev <
> > > > hdfs-...@hadoop.apache.org>; yarn-dev ;
> > > > mapreduce-dev 
> > > > 主题: [VOTE] Release Apache Hadoop 3.3.6 RC1
> > > >
> > > > I am inviting anyone to try and vote on this release candidate.
> > > >
> > > > Note:
> > > > This is exactly the same as RC0, except the CHANGELOG.
> > > >
> > > > The RC is available at:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> > > > https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
> > > >
> > > > Git tag: release-3.3.6-RC1
> > > > https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
> > > >
> > > > Maven artifacts is built by x86 machine and are staged at
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
> > > >
> > > > My public key:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Changelog:
> > > > https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
> > > >
> > > > Release notes:
> > > >
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
> > > >
> > > > This is a relatively small release (by Hadoop standard) containing
> about
> > > > 120 commits.
> > > > Please give it a try, this RC vote will run for 7 days.
> > > >
> > > >
> > > > Feature highlights:
> > > >
> > > > SBOM artifacts
> > > > 
> > > > Starting from this release, Hadoop publishes Software Bill of
> Materials
> > > > (SBOM) using
> > > > CycloneDX Maven plugin. For more information about SBOM, please go to
> > > > [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
> > > >
> > > > HDFS RBF: RDBMS based token storage support
> > > > 
> > > > HDFS Router-Router Based Federation now 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Ayush Saxena]

Folks you had put your vote on the wrong thread, this is the RC0 thead, you 
need to put on the RC1,
https://lists.apache.org/thread/022p4rml5tvsx9xpq6t7b3n1td8lzz1d

-Ayush

Sent from my iPhone

> On 20-Jun-2023, at 10:03 PM, Ashutosh Gupta  
> wrote:
> 
> Hi
> 
> Thanks Wei-Chiu for driving the release.
> 
> +1 (non-binding)
> 
> * Builds from source looks good.
> * Checksums and signatures look good.
> * Running basic HDFS commands and running simple MapReduce jobs looks good.
> * hadoop-tools/hadoop-aws UTs and ITs looks good
> 
> Thanks,
> Ash
> 
>> On Tue, Jun 20, 2023 at 5:18 PM Mukund Madhav Thakur
>>  wrote:
>> 
>> Hi Wei-Chiu,
>> Thanks for driving the release.
>> 
>> +1 (binding)
>> Verified checksum and signature.
>> Built from source successfully.
>> Ran aws Itests
>> Ran azure Itests
>> Compiled hadoop-api-shim
>> Compiled google cloud storage.
>> 
>> 
>> I did see the two test failures in GCS connector as well but those are
>> harmless.
>> 
>> 
>> 
>> On Thu, Jun 15, 2023 at 8:21 PM Wei-Chiu Chuang
>>  wrote:
>> 
>>> Overall so far so good.
>>> 
>>> hadoop-api-shim:
>>> built, tested successfully.
>>> 
>>> cloudstore:
>>> built successfully.
>>> 
>>> Spark:
>>> built successfully. Passed hadoop-cloud tests.
>>> 
>>> Ozone:
>>> One test failure due to unrelated Ozone issue. This test is being
>> disabled
>>> in the latest Ozone code.
>>> 
>>> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
>>> to load library ozone_rocksdb_tools from both java.library.path &
>>> resource file libozone_rocksdb_t
>>> ools.so from jar.
>>>at
>>> 
>> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
>>> 
>>> 
>>> Google gcs:
>>> There are two test failures. The tests were added recently by
>> HADOOP-18724
>>>  in Hadoop 3.3.6.
>> This
>>> is okay. Not production code problem. Can be addressed in GCS code.
>>> 
>>> [ERROR] Errors:
>>> [ERROR]
>>> 
>>> 
>> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
>>> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
>>> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
>>> [ERROR]
>>> 
>>> 
>> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
>>> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
>>> ucket/contract-test/testOpenFileApplyAsyncRead
>> "fs.option.openfile.length"
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
>>> wrote:
>>> 
 The hbase-filesystem tests passed after reverting HADOOP-18596
  and HADOOP-18633
  from my local
>> tree.
 So I think it's a matter of the default behavior being changed. It's
>> not
 the end of the world. I think we can address it by adding an
>> incompatible
 change flag and a release note.
 
 On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
 wrote:
 
> Cross referenced git history and jira. Changelog needs some update
> 
> Not in the release
> 
>   1. HDFS-16858 
> 
> 
>   1. HADOOP-18532 <
>> https://issues.apache.org/jira/browse/HADOOP-18532>
>   2.
>  1. HDFS-16861 >> 
> 2.
>1. HDFS-16866
>
>2.
>   1. HADOOP-18320
>   
>   2.
> 
> Updated fixed version. Will generate. new Changelog in the next RC.
> 
> Was able to build HBase and hbase-filesystem without any code change.
> 
> hbase has one unit test failure. This one is reproducible even with
> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> 
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
>> elapsed:
> 9.007 s <<< FAILURE! - in
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> [ERROR]
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> Time elapsed: 3.13 s  <<< ERROR!
> java.lang.OutOfMemoryError: Java heap space
> at
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> at
> 
>>> 
>> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> 
> hbase-filesystem has three test failures in TestHBOSSContractDistCp,
>> and
> is not reproducible with Hadoop 3.3.5.
> [ERROR] Failures: [ERROR]
> 
>>> 
>> 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Ayush Saxena]

Hi Masatake,
That ticket is just a backport of:
https://issues.apache.org/jira/browse/HADOOP-17612 and the only java code
change is in test [1], which is causing issues for you. That ticket isn't
marked incompatible either, so maybe that is why it got pulled back.

btw. the original PR does mention it to be compatible [2].

Are you trying to compile ZK-3.5 with Hadoop-3.3.6 or Hadoop-3.3.6 with
ZK-3.5, If later and then if the compilation fails, then it shouldn't be an
incompatible change, right? or do we need to maintain compat that way as
well?

+1 in maintaining compatibility, incompatible changes should be avoided as
far as possible unless excessively necessary even in trunk or like we need
to do it for some "relevant" security issue or so in those thirdparty libs.
The RC1 vote is already up, do you plan to get this change excluded from
that?

Regarding the test, If you pass ``null`` instead of that DisconnectReason,
then also the test test passes, but I am pretty sure you would get
NoSuchMethod error for closeAll after that because that closeAll ain't
there is Zk-3.5, ZOOKEEPER-3439 removed it

-Ayush

PS. From this doc: https://zookeeper.apache.org/releases.html, even ZK-3.6
line is also EOL, not sure how those guys operate :-)

[1]
https://github.com/apache/hadoop/pull/3241/files#diff-b273546d6f060e617553eaa49da69039d2c655a77d42022779c2281d0f6cd08eR135
[2] https://github.com/apache/hadoop/pull/3241#issuecomment-889185103

On Mon, 19 Jun 2023 at 06:44, Masatake Iwasaki 
wrote:

> I got compilation error against ZooKeeper 3.5 due to HADOOP-18515.
> It should be marked as incompatible change?
> https://issues.apache.org/jira/browse/HADOOP-18515
>
> ::
>
>[ERROR]
> /home/rocky/srcs/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.6-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverControllerStress.java:[135,40]
> cannot find symbol
>  symbol:   variable DisconnectReason
>  location: class org.apache.zookeeper.server.ServerCnxn
>
> While ZooKeeper 3.5 is already EoL, it would be nice to keep compatibility
> in a patch release
> especially if only test code is the cause.
>
> Thanks,
> Masatake Iwasaki
>
> On 2023/06/18 4:57, Wei-Chiu Chuang wrote:
> > I was going to do another RC in case something comes up.
> > But it looks like the only thing that needs to be fixed is the Changelog.
> >
> >
> > 1. HADOOP-18596 
> >
> > HADOOP-18633 
> > are related to cloud store semantics, and I don't want to make a
> judgement
> > call on it. As far as I can tell its effect can be addressed by
> supplying a
> > config option in the application code.
> > It looks like the feature improves fault tolerance by ensuring files are
> > synchronized if modification time is different between the source and
> > destination. So to me it's the better behavior.
> >
> > I can make a RC1 over the weekend to fix the Changelog but that's
> probably
> > the only thing that's going to have.
> > On Sat, Jun 17, 2023 at 2:00 AM Xiaoqiao He 
> wrote:
> >
> >> Thanks Wei-Chiu for driving this release. The next RC will be prepared,
> >> right?
> >> If true, I would like to try and vote on the next RC.
> >> Just notice that some JIRAs are not included and need to revert some
> PRs to
> >> pass HBase verification which are mentioned above.
> >>
> >> Best Regards,
> >> - He Xiaoqiao
> >>
> >>
> >> On Fri, Jun 16, 2023 at 9:20 AM Wei-Chiu Chuang
> >>  wrote:
> >>
> >>> Overall so far so good.
> >>>
> >>> hadoop-api-shim:
> >>> built, tested successfully.
> >>>
> >>> cloudstore:
> >>> built successfully.
> >>>
> >>> Spark:
> >>> built successfully. Passed hadoop-cloud tests.
> >>>
> >>> Ozone:
> >>> One test failure due to unrelated Ozone issue. This test is being
> >> disabled
> >>> in the latest Ozone code.
> >>>
> >>> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> >>> to load library ozone_rocksdb_tools from both java.library.path &
> >>> resource file libozone_rocksdb_t
> >>> ools.so from jar.
> >>>  at
> >>>
> >>
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
> >>>
> >>>
> >>> Google gcs:
> >>> There are two test failures. The tests were added recently by
> >> HADOOP-18724
> >>>  in Hadoop 3.3.6.
> >> This
> >>> is okay. Not production code problem. Can be addressed in GCS code.
> >>>
> >>> [ERROR] Errors:
> >>> [ERROR]
> >>>
> >>>
> >>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> >>> » IllegalArgument Unknown mandatory key for
> gs://fake-in-memory-test-buck
> >>> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> >>> [ERROR]
> >>>
> >>>
> >>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> >>> » IllegalArgument Unknown mandatory key for 

[jira] [Resolved] (YARN-11506) The formatted yarn queue list is displayed on the command line

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11506.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> The formatted yarn queue list is displayed on the command line
> --
>
> Key: YARN-11506
> URL: https://issues.apache.org/jira/browse/YARN-11506
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Lu Yuan
>Assignee: Lu Yuan
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> h4. In the issue list YARN-11474, the printed information is not aligned,This 
> issue fixes the yarn queue display formatting problem



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Fwd: Call for Presentations, Community Over Code Asia 2023

[Ayush Saxena]

Forwarding from dev@hadoop to the ML we use.
Original Mail:
https://lists.apache.org/thread/3c16xyvmdyvg4slmtfywcwjbck1b4x02

-Ayush

-- Forwarded message -
From: Rich Bowen 
Date: Mon, 5 Jun 2023 at 21:43
Subject: Call for Presentations, Community Over Code Asia 2023
To: rbo...@apache.org 


You are receiving this message because you are subscribed to one more
more developer mailing lists at the Apache Software Foundation.

The call for presentations is now open at
"https://apachecon.com/acasia2023/cfp.html;, and will be closed by
Sunday, Jun 18th, 2023 11:59 PM GMT.

The event will be held in Beijing, China, August 18-20, 2023.

We are looking for presentations about anything relating to Apache
Software Foundation projects, open-source governance, community, and
software development.
In particular, this year we are building content tracks around the
following specific topics/projects:

AI / Machine learning
API / Microservice
Community
CloudNative
Data Storage & Computing
DataOps
Data Lake & Data Warehouse
OLAP & Data Analysis
Performance Engineering
Incubator
IoT/IIoT
Messaging
RPC
Streaming
Workflow / Data Processing
Web Server / Tomcat

If your proposed presentation falls into one of these categories,
please select that topic in the CFP entry form. Or select Others if
it’s related to another topic or project area.

Looking forward to hearing from you!

Willem Jiang, and the Community Over Code planners


-
To unsubscribe, e-mail: dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11492) Improve createJerseyClient#setConnectTimeout Code

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11492.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Improve createJerseyClient#setConnectTimeout Code
> -
>
> Key: YARN-11492
> URL: https://issues.apache.org/jira/browse/YARN-11492
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> createJerseyClient#setConnectTimeout There is a type conversion code that 
> converts long to int, we should be careful with this part of code, I added 
> some verification rules.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Issues with Github PR integration with Jira

[Ayush Saxena]

It is sorted now. the github bot got blocked as a spam user. poor fellow
:-)

-Ayush

On Fri, 19 May 2023 at 05:33, Ayush Saxena  wrote:

> Hi Folks,
> Just wanted to let everyone know, there is an ongoing issue with
> Github PR & Jira integration since the last few days.
>
> The Pull Requests aren't getting linked automatically to Jira in some
> cases, If you observe that Please link manually unless this issue is
> resolved.
>
> I have an INFRA ticket in place for it [1] and I have validated the
> yaml files are intact. [2], so it shouldn't be Hadoop specific
>
>
> -Ayush
>
> [1] https://issues.apache.org/jira/browse/INFRA-24608
> [2] https://github.com/apache/hadoop/blame/trunk/.asf.yaml#L27
>

[jira] [Resolved] (YARN-11496) Improve TimelineService log format

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11496.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Improve TimelineService log format
> --
>
> Key: YARN-11496
> URL: https://issues.apache.org/jira/browse/YARN-11496
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: timelineservice
>Affects Versions: 3.3.4
>Reporter: Xianming Lei
>Assignee: Xianming Lei
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Call for Presentations, Community Over Code 2023

[Ayush Saxena]

Thanx Wei-Chiu,

By any chance if you have access to the presentations, do share!!!

Was looking at the Apache Iceberg website[1], they maintain a list of
all the blogs shared anywhere, was thinking if we can collect some
blogs/youtube links around some fascinating talks and have such a page
on our website as well. Obviously if people feel good about it, just
an idea that came into my mind looking at the iceberg website and
thought of sharing.


[1] https://iceberg.apache.org/blogs/

On Wed, 10 May 2023 at 10:13, Wei-Chiu Chuang  wrote:
>
> There's also a call for presentation for Community over Code Asia 2023
>
> https://www.bagevent.com/event/cocasia-2023-EN
> Happening Aug 18-20. CfP due by 6/6
>
>
> On Tue, May 9, 2023 at 8:39 PM Ayush Saxena  wrote:
>>
>> Forwarding from dev@hadoop to the dev ML which we use.
>>
>> The actual mail lies here:
>> https://www.mail-archive.com/dev@hadoop.apache.org/msg00160.html
>>
>> -Ayush
>>
>> On 2023/05/09 21:24:09 Rich Bowen wrote:
>> > (Note: You are receiving this because you are subscribed to the dev@
>> > list for one or more Apache Software Foundation projects.)
>> >
>> > The Call for Presentations (CFP) for Community Over Code (formerly
>> > Apachecon) 2023 is open at
>> > https://communityovercode.org/call-for-presentations/, and will close
>> > Thu, 13 Jul 2023 23:59:59 GMT.
>> >
>> > The event will be held in Halifax, Canada, October 7-10, 2023.
>> >
>> > We welcome submissions on any topic related to the Apache Software
>> > Foundation, Apache projects, or the communities around those projects.
>> > We are specifically looking for presentations in the following
>> > catetegories:
>> >
>> > Fintech
>> > Search
>> > Big Data, Storage
>> > Big Data, Compute
>> > Internet of Things
>> > Groovy
>> > Incubator
>> > Community
>> > Data Engineering
>> > Performance Engineering
>> > Geospatial
>> > API/Microservices
>> > Frameworks
>> > Content Wrangling
>> > Tomcat and httpd
>> > Cloud and Runtime
>> > Streaming
>> > Sustainability
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>> Sent from my iPhone
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Issues with Github PR integration with Jira

[Ayush Saxena]

Hi Folks,
Just wanted to let everyone know, there is an ongoing issue with
Github PR & Jira integration since the last few days.

The Pull Requests aren't getting linked automatically to Jira in some
cases, If you observe that Please link manually unless this issue is
resolved.

I have an INFRA ticket in place for it [1] and I have validated the
yaml files are intact. [2], so it shouldn't be Hadoop specific


-Ayush

[1] https://issues.apache.org/jira/browse/INFRA-24608
[2] https://github.com/apache/hadoop/blame/trunk/.asf.yaml#L27

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11495) Fix typos in hadoop-yarn-server-web-proxy

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11495.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix typos in hadoop-yarn-server-web-proxy
> -
>
> Key: YARN-11495
> URL: https://issues.apache.org/jira/browse/YARN-11495
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: webapp
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Minor
> Fix For: 3.4.0
>
>
> While reading the code, I found some typo issues. This JIRA will fix these 
> issues in hadoop-yarn-server-web-proxy.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

RE: Call for Presentations, Community Over Code 2023

[Ayush Saxena]

Forwarding from dev@hadoop to the dev ML which we use.

The actual mail lies here:
https://www.mail-archive.com/dev@hadoop.apache.org/msg00160.html

-Ayush

On 2023/05/09 21:24:09 Rich Bowen wrote:
> (Note: You are receiving this because you are subscribed to the dev@
> list for one or more Apache Software Foundation projects.)
>
> The Call for Presentations (CFP) for Community Over Code (formerly
> Apachecon) 2023 is open at
> https://communityovercode.org/call-for-presentations/, and will close
> Thu, 13 Jul 2023 23:59:59 GMT.
>
> The event will be held in Halifax, Canada, October 7-10, 2023.
>
> We welcome submissions on any topic related to the Apache Software
> Foundation, Apache projects, or the communities around those projects.
> We are specifically looking for presentations in the following
> catetegories:
>
> Fintech
> Search
> Big Data, Storage
> Big Data, Compute
> Internet of Things
> Groovy
> Incubator
> Community
> Data Engineering
> Performance Engineering
> Geospatial
> API/Microservices
> Frameworks
> Content Wrangling
> Tomcat and httpd
> Cloud and Runtime
> Streaming
> Sustainability
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: dev-h...@hadoop.apache.org
>
>


Sent from my iPhone

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

One thing I forgot to mention. If you are planning to run tests also for
Windows OS then github actions isn’t something to be chased. It has 6 hrs
timeout.

So, in case of multi module changes or root level changes, we will get
screwed, the tests take more than 20 hrs in those cases, so we will hit the
timeout

-Ayush

On Tue, 9 May 2023 at 11:49 PM, Ayush Saxena  wrote:

> Hi Gautham,
> The only advantage I can think of is your limitation to nodes shouldn’t
> bother much. Though there are limited resources given to Apache but still
> never observed any starvation in any project for the resources using Github
> Actions.
>
> If Precommit CI for windows is there and it can work in parallel, that
> should be good as well
>
> Good Luck!!!
>
> -Ayush
>
> On Tue, 9 May 2023 at 8:47 PM, Gautham Banasandra 
> wrote:
>
>> Hi Ayush,
>>
>> I was planning to set up the precommit CI job for Windows 10, so that
>> we validate each PR against Windows 10. The precommit CI can actually
>> run in parallel to the Linux precommit CI. But I think we've got very few
>> Windows nodes (about 12 I think) compared to the number of Linux nodes
>> (about 40 in number).
>> However, I think the setting up the precommit CI is quite worth
>> the effort.
>> It was quite hard to get to this point and we certainly don't want to
>> regress
>> in this regard. I think I'll pursue this once I stabilize the nightly CI
>> and after
>> I sort out the rest of the failing parts (mvnsite, javadoc etc).
>>
>> Also, does Github Actions provide any advantage over precommit CI?
>>
>> Thanks,
>> --Gautham
>>
>> On Tue, 9 May 2023 at 02:44, Ayush Saxena  wrote:
>>
>>> Awesome!!!
>>>
>>> Do you plan to run some builds per PR to ensure no commits breaks it
>>> in the future? Worth exploring GithubActions in the later stages if we
>>> can have an action for maven install on windows, it won't affect the
>>> normal build time as well. can run in parallel with the existing
>>> builds.
>>>
>>> -Ayush
>>>
>>> On Mon, 8 May 2023 at 23:39, Gautham Banasandra 
>>> wrote:
>>> >
>>> > Dear Hadoop community,
>>> >
>>> > It is my pleasure to announce that I've set up the Nightly Jenkins CI
>>> for
>>> > Hadoop on the Windows 10 platform[1]. The effort mainly involved
>>> getting
>>> > Yetus to run on Windows against Hadoop.
>>> > The nightly CI will run every 36 hours and send out the build report
>>> to the
>>> > same recipients as this email, upon completion.
>>> > There are still quite a few things that need to be sorted out.
>>> Currently,
>>> > mvninstall has a +1. Other phases like mvnsite, javadoc etc still need
>>> to be
>>> > fixed.
>>> >
>>> > [1]
>>> >
>>> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>>> >
>>> > Thanks,
>>> > --Gautham
>>>
>>

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

Hi Gautham,
The only advantage I can think of is your limitation to nodes shouldn’t
bother much. Though there are limited resources given to Apache but still
never observed any starvation in any project for the resources using Github
Actions.

If Precommit CI for windows is there and it can work in parallel, that
should be good as well

Good Luck!!!

-Ayush

On Tue, 9 May 2023 at 8:47 PM, Gautham Banasandra 
wrote:

> Hi Ayush,
>
> I was planning to set up the precommit CI job for Windows 10, so that
> we validate each PR against Windows 10. The precommit CI can actually
> run in parallel to the Linux precommit CI. But I think we've got very few
> Windows nodes (about 12 I think) compared to the number of Linux nodes
> (about 40 in number).
> However, I think the setting up the precommit CI is quite worth the effort.
> It was quite hard to get to this point and we certainly don't want to
> regress
> in this regard. I think I'll pursue this once I stabilize the nightly CI
> and after
> I sort out the rest of the failing parts (mvnsite, javadoc etc).
>
> Also, does Github Actions provide any advantage over precommit CI?
>
> Thanks,
> --Gautham
>
> On Tue, 9 May 2023 at 02:44, Ayush Saxena  wrote:
>
>> Awesome!!!
>>
>> Do you plan to run some builds per PR to ensure no commits breaks it
>> in the future? Worth exploring GithubActions in the later stages if we
>> can have an action for maven install on windows, it won't affect the
>> normal build time as well. can run in parallel with the existing
>> builds.
>>
>> -Ayush
>>
>> On Mon, 8 May 2023 at 23:39, Gautham Banasandra 
>> wrote:
>> >
>> > Dear Hadoop community,
>> >
>> > It is my pleasure to announce that I've set up the Nightly Jenkins CI
>> for
>> > Hadoop on the Windows 10 platform[1]. The effort mainly involved getting
>> > Yetus to run on Windows against Hadoop.
>> > The nightly CI will run every 36 hours and send out the build report to
>> the
>> > same recipients as this email, upon completion.
>> > There are still quite a few things that need to be sorted out.
>> Currently,
>> > mvninstall has a +1. Other phases like mvnsite, javadoc etc still need
>> to be
>> > fixed.
>> >
>> > [1]
>> >
>> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>> >
>> > Thanks,
>> > --Gautham
>>
>

Re: Nightly Jenkins CI for Hadoop on Windows 10

[Ayush Saxena]

Awesome!!!

Do you plan to run some builds per PR to ensure no commits breaks it
in the future? Worth exploring GithubActions in the later stages if we
can have an action for maven install on windows, it won't affect the
normal build time as well. can run in parallel with the existing
builds.

-Ayush

On Mon, 8 May 2023 at 23:39, Gautham Banasandra  wrote:
>
> Dear Hadoop community,
>
> It is my pleasure to announce that I've set up the Nightly Jenkins CI for
> Hadoop on the Windows 10 platform[1]. The effort mainly involved getting
> Yetus to run on Windows against Hadoop.
> The nightly CI will run every 36 hours and send out the build report to the
> same recipients as this email, upon completion.
> There are still quite a few things that need to be sorted out. Currently,
> mvninstall has a +1. Other phases like mvnsite, javadoc etc still need to be
> fixed.
>
> [1]
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-win10-x86_64/
>
> Thanks,
> --Gautham

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.6 release planning

[Ayush Saxena]

That openssl change ain't a blocker now from my side, that ABFS-Jdk-17
stuff got sorted out, Steve knew a way out

On Sat, 6 May 2023 at 00:51, Ayush Saxena  wrote:
>
> Thanx Wei-Chiu for the initiative, Good to have quick releases :)
>
> With my Hive developer hat on, I would like to bring some stuff up for
> consideration(feel free to say no, if it is beyond scope or feels even
> a bit unsafe, don't want to mess up the release)
>
> * HADOOP-18662: ListFiles with recursive fails with FNF : This broke
> compaction in Hive, bothers only with HDFS though. There is a
> workaround to that, if it doesn't feel safe. no issues, or if some
> improvements suggested. I can quickly do that :)
>
> * HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
> but if it works and is safe then to 2.2.5. I got flagged today that
> this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
> think(need to dig in more),
>
> Now for the dependency upgrades:
>
> A big NO to Jackson, that ain't safe and the wounds are still fresh,
> it screwed the 3.3.3 release for many projects. So, let's not get into
> that. Infact anything that touches those shaded jars is risky, some
> package-json exclusion also created a mess recently. So, Lets not
> touch only and that too when we have less time.
>
> Avoid anything around Jetty upgrade, I have selfish reasons for that.
> Jetty messes something up with Hbase and Hive has a dependency on
> Hbase, and it is crazy, in case interested [1]. So, any upgrade to
> Jetty will block hive from upgrading Hadoop as of today. But that is a
> selfish reason and just up for consideration. Go ahead if necessary. I
> just wanted to let folks know
>
>
> Apart from the Jackson stuff, everything is suggestive in nature, your
> call feel free to ignore.
>
> @Xiaoqiao He , maybe pulling in all those 100+ would be risky,
> considering the timelines, but if we find a few fancy safe tickets,
> maybe if you have identified any already, can put them up on this
> thread and if folks are convinced. We can get them in? Juzz my
> thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)
>
>
> God Luck
>
> -Ayush
>
> [1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803
>
> On Fri, 5 May 2023 at 16:13, Steve Loughran  
> wrote:
> >
> > Wei-Chiu has suggested a minimal "things in 3.3.5 which were very broken,
> > api change for ozone and any critical jar updates"
> >
> > so much lower risk/easier to qualify and ship.
> >
> > I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in here;
> > maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> > remove json-smart...)
> >
> > I'd also like to downgrade protobuf 2.5 from required to optional; even
> > though hadoop uses the shaded one, to support hbase etc the IPC code still
> > has direct use of the 2.5 classes. that coud be optional
> >
> > if anyone wants to take up this PR, I would be very happy
> > https://github.com/apache/hadoop/pull/4996
> >
> > On Fri, 5 May 2023 at 04:27, Xiaoqiao He  wrote:
> >
> > > Thanks Wei-Chiu for driving this release.
> > > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed version.
> > >
> > > so far only 8 jiras were resolved in the branch-3.3 line.
> > >
> > >
> > > If we should consider both 3.3.6 and 3.3.9 (which is from release-3.3.5
> > > discuss)[1] for this release line?
> > > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > > fixVersion in (3.3.6, 3.3.9)`[2],
> > > there are more than hundred jiras now.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > > [2]
> > >
> > > https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> > >
> > >
> > > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang  wrote:
> > >
> > > > Hi community,
> > > >
> > > > I'd like to kick off the discussion around Hadoop 3.3.6 release plan.
> > > >
> > > > I'm being selfish but my intent for 3.3.6 is to have the new APIs in
> > > > HADOOP-18671 <https://issues.apache.org/jira/browse/HADOOP-18671> added
> > > so
> > > > we can have HBase to adopt this new API. Other than that, perhaps
> > > > thirdparty dependency updates.
> > > >
> > > > I

Re: [DISCUSS] Hadoop 3.3.6 release planning

[Ayush Saxena]

Thanx Wei-Chiu for the initiative, Good to have quick releases :)

With my Hive developer hat on, I would like to bring some stuff up for
consideration(feel free to say no, if it is beyond scope or feels even
a bit unsafe, don't want to mess up the release)

* HADOOP-18662: ListFiles with recursive fails with FNF : This broke
compaction in Hive, bothers only with HDFS though. There is a
workaround to that, if it doesn't feel safe. no issues, or if some
improvements suggested. I can quickly do that :)

* HADOOP-17649: Update wildfly openssl to 2.1.3.Final. Maybe not 2.1.3
but if it works and is safe then to 2.2.5. I got flagged today that
this openssl creates a bit of mess with JDK-17 for Hive with ABFS I
think(need to dig in more),

Now for the dependency upgrades:

A big NO to Jackson, that ain't safe and the wounds are still fresh,
it screwed the 3.3.3 release for many projects. So, let's not get into
that. Infact anything that touches those shaded jars is risky, some
package-json exclusion also created a mess recently. So, Lets not
touch only and that too when we have less time.

Avoid anything around Jetty upgrade, I have selfish reasons for that.
Jetty messes something up with Hbase and Hive has a dependency on
Hbase, and it is crazy, in case interested [1]. So, any upgrade to
Jetty will block hive from upgrading Hadoop as of today. But that is a
selfish reason and just up for consideration. Go ahead if necessary. I
just wanted to let folks know


Apart from the Jackson stuff, everything is suggestive in nature, your
call feel free to ignore.

@Xiaoqiao He , maybe pulling in all those 100+ would be risky,
considering the timelines, but if we find a few fancy safe tickets,
maybe if you have identified any already, can put them up on this
thread and if folks are convinced. We can get them in? Juzz my
thoughts, it is up to you and Wei-Chiu, (No skin in the game opinion)


God Luck

-Ayush

[1] https://github.com/apache/hive/pull/4290#issuecomment-1536553803

On Fri, 5 May 2023 at 16:13, Steve Loughran  wrote:
>
> Wei-Chiu has suggested a minimal "things in 3.3.5 which were very broken,
> api change for ozone and any critical jar updates"
>
> so much lower risk/easier to qualify and ship.
>
> I need to get https://issues.apache.org/jira/browse/HADOOP-18724 in here;
> maybe look at a refresh of the "classic" jars (slf4j, reload, jackson*,
> remove json-smart...)
>
> I'd also like to downgrade protobuf 2.5 from required to optional; even
> though hadoop uses the shaded one, to support hbase etc the IPC code still
> has direct use of the 2.5 classes. that coud be optional
>
> if anyone wants to take up this PR, I would be very happy
> https://github.com/apache/hadoop/pull/4996
>
> On Fri, 5 May 2023 at 04:27, Xiaoqiao He  wrote:
>
> > Thanks Wei-Chiu for driving this release.
> > Cherry-pick YARN-11482 to branch-3.3 and mark 3.3.6 as the fixed version.
> >
> > so far only 8 jiras were resolved in the branch-3.3 line.
> >
> >
> > If we should consider both 3.3.6 and 3.3.9 (which is from release-3.3.5
> > discuss)[1] for this release line?
> > I try to query with `project in (HDFS, YARN, HADOOP, MAPREDUCE) AND
> > fixVersion in (3.3.6, 3.3.9)`[2],
> > there are more than hundred jiras now.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > [1] https://lists.apache.org/thread/kln96frt2tcg93x6ht99yck9m7r9qwxp
> > [2]
> >
> > https://issues.apache.org/jira/browse/YARN-11482?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20fixVersion%20in%20(3.3.6%2C%203.3.9)
> >
> >
> > On Fri, May 5, 2023 at 1:19 AM Wei-Chiu Chuang  wrote:
> >
> > > Hi community,
> > >
> > > I'd like to kick off the discussion around Hadoop 3.3.6 release plan.
> > >
> > > I'm being selfish but my intent for 3.3.6 is to have the new APIs in
> > > HADOOP-18671  added
> > so
> > > we can have HBase to adopt this new API. Other than that, perhaps
> > > thirdparty dependency updates.
> > >
> > > If you have open items to be added in the coming weeks, please add 3.3.6
> > to
> > > the target release version. Right now I am only seeing three open jiras
> > > targeting 3.3.6.
> > >
> > > I imagine this is going to be a small release as 3.3.5 (hat tip to Steve)
> > > was only made two months back, and so far only 8 jiras were resolved in
> > the
> > > branch-3.3 line.
> > >
> > > Best,
> > > Weichiu
> > >
> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

We have a daily build running for 3.3.5:
https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/

We have already released it, so I feel we can disable it. Will do it
tomorrow, if nobody objects. In case the one who configured it wants
to do it early, feel free to do so.

We already have one for branch-3.3 which runs weekly which most
probably most of us don't follow :)

-Ayush

On Wed, 22 Mar 2023 at 00:20, Steve Loughran
 wrote:
>
> ok, here's my summary, even though most of the binding voters forgot to
> declare they were on the PMC.
>
> +1 binding
>
> Steve Loughran
> Chris Nauroth
> Masatake Iwasaki
> Ayush Saxena
> Xiaoqiao He
>
> +1 non-binding
>
> Viraj Jasani
>
>
> 0 or -1 votes: none.
>
>
> Accordingly: the release is good!
>
> I will send the formal announcement out tomorrow
>
> A big thank you to everyone who qualified the RC, I know its a lot of work.
> We can now get this out and *someone else* can plan the followup.
>
>
> steve
>
> On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:
>
> > +1
> >
> > Thank you for the release candidate, Steve!
> >
> > * Verified all checksums.
> > * Verified all signatures.
> > * Built from source, including native code on Linux.
> > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> > -Drequire.zstd -DskipTests
> > * Tests passed.
> > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > * Checked dependency tree to make sure we have all of the expected library
> > updates that are mentioned in the release notes.
> > * mvn -o dependency:tree
> > * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> > with no code.
> > * For ARM verification:
> > * Ran "file " on all native binaries in the ARM tarball to confirm
> > they actually came out with ARM as the architecture.
> > * Output of hadoop checknative -a on ARM looks good.
> > * Ran a MapReduce job with the native bzip2 codec for compression, and
> > it worked fine.
> > * Ran a MapReduce job with YARN configured to use
> > LinuxContainerExecutor and verified launching the containers through
> > container-executor worked.
> >
> > Chris Nauroth
> >
> >
> > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena  wrote:
> >
> > > +1(Binding)
> > >
> > > * Built from source (x86 & ARM)
> > > * Successful Native Build (x86 & ARM)
> > > * Verified Checksums (x86 & ARM)
> > > * Verified Signature (x86 & ARM)
> > > * Checked the output of hadoop version (x86 & ARM)
> > > * Verified the output of hadoop checknative (x86 & ARM)
> > > * Ran some basic HDFS shell commands.
> > > * Ran some basic Yarn shell commands.
> > > * Played a bit with HDFS Erasure Coding.
> > > * Ran TeraGen & TeraSort
> > > * Browed through NN, DN, RM & NM UI
> > > * Skimmed over the contents of website.
> > > * Skimmed over the contents of maven repo.
> > > * Selectively ran some HDFS & CloudStore tests
> > >
> > > Thanx Steve for driving the release. Good Luck!!!
> > >
> > > -Ayush
> > >
> > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> > wrote:
> > > >
> > > > +1
> > > >
> > > > * Verified signature and checksum of the source tarball.
> > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > * Setup pseudo cluster with HDFS and YARN.
> > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> > > > result.
> > > > * Run example mr applications and check the result - Pi & wordcount.
> > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> > etc.
> > > >
> > > > Thanks Steve for your work.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> > > iwasak...@oss.nttdata.com>
> > > >> wrote:
> > > >>
> > > >> +1
> > > >>
> > > >> + verified the signature and checksum of the source tarball.
> > > >>
> > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK
> > 

A Message from the Board to PMC members

[Ayush Saxena]

 d...@hadoop.apache.org received a mail from the apache board,
forwarding it to the mailing lists which we use, sharing the link
here:

https://www.mail-archive.com/dev@hadoop.apache.org/msg00158.html

private@ in bcc

-Ayush

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

[Ayush Saxena]

>
>  it's already hard to migrate from JDK8 why not retarget JDK17.
>

+1, makes sense to me, sounds like a win-win situation to me, though there
would be some additional issues to chase now :)

-Ayush


On Tue, 28 Mar 2023 at 23:29, Wei-Chiu Chuang  wrote:

> My random thoughts. Probably bad takes:
>
> There are projects experimenting with JDK17 now.
> JDK11 active support will end in 6 months. If it's already hard to migrate
> from JDK8 why not retarget JDK17.
>
> On Tue, Mar 28, 2023 at 10:30 AM Ayush Saxena  wrote:
>
>> I know Jersey upgrade as a blocker. Some folks were chasing that last
>> year during 3.3.4 time, I don’t know where it is now, didn’t see then
>> what’s the problem there but I remember there was some intitial PR which
>> did it for HDFS atleast, so I never looked beyond that…
>>
>> I too had jdk-11 in my mind, but only for trunk. 3.4.x can stay as
>> java-11 only branch may be, but that is something later to decide, once we
>> get the code sorted…
>>
>> -Ayush
>>
>> > On 28-Mar-2023, at 9:16 PM, Steve Loughran 
>> wrote:
>> >
>> > well, how about we flip the switch and get on with it.
>> >
>> > slf4j seems happy on java11,
>> >
>> > side issue, anyone seen test failures on zulu1.8; somehow my test run is
>> > failing and i'm trying to work out whether its a mismatch in command
>> > line/ide jvm versions, or the 3.3.5 JARs have been built with an openjdk
>> > version which requires IntBuffer implements an overridden method
>> IntBuffer
>> > rewind().
>> >
>> > java.lang.NoSuchMethodError:
>> java.nio.IntBuffer.rewind()Ljava/nio/IntBuffer;
>> >
>> > at
>> org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:341)
>> > at
>> >
>> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:308)
>> > at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:257)
>> > at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:202)
>> > at java.io.DataInputStream.read(DataInputStream.java:149)
>> >
>> >> On Tue, 28 Mar 2023 at 15:52, Viraj Jasani  wrote:
>> >> IIRC some of the ongoing major dependency upgrades (log4j 1 to 2,
>> jersey 1
>> >> to 2 and junit 4 to 5) are blockers for java 11 compile + test
>> stability.
>> >> On Tue, Mar 28, 2023 at 4:55 AM Steve Loughran
>> > >> wrote:
>> >>> Now that hadoop 3.3.5 is out, i want to propose something new
>> >>> we switch branch-3.3 and trunk to being java11 only
>> >>> 1. java 11 has been out for years
>> >>> 2. oracle java 8 is no longer available under "premier support"; you
>> >>> can't really get upgrades
>> >>> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>> >>> 3. openJDK 8 releases != oracle ones, and things you compile with them
>> >>> don't always link to oracle java 8 (some classes in java.nio have
>> >> added
>> >>> more overrides)
>> >>> 4. more and more libraries we want to upgrade to/bundle are java 11
>> >> only
>> >>> 5. moving to java 11 would cut our yetus build workload in half, and
>> >>> line up for adding java 17 builds instead.
>> >>> I know there are some outstanding issues still in
>> >>> https://issues.apache.org/jira/browse/HADOOP-16795 -but are they
>> >> blockers?
>> >>> Could we just move to java11 and enhance at our leisure, once java8
>> is no
>> >>> longer a concern.
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>

Re: [DISCUSS] hadoop branch-3.3+ going to java11 only

[Ayush Saxena]

I know Jersey upgrade as a blocker. Some folks were chasing that last year 
during 3.3.4 time, I don’t know where it is now, didn’t see then what’s the 
problem there but I remember there was some intitial PR which did it for HDFS 
atleast, so I never looked beyond that…

I too had jdk-11 in my mind, but only for trunk. 3.4.x can stay as java-11 only 
branch may be, but that is something later to decide, once we get the code 
sorted…

-Ayush

> On 28-Mar-2023, at 9:16 PM, Steve Loughran  
> wrote:
> 
> well, how about we flip the switch and get on with it.
> 
> slf4j seems happy on java11,
> 
> side issue, anyone seen test failures on zulu1.8; somehow my test run is
> failing and i'm trying to work out whether its a mismatch in command
> line/ide jvm versions, or the 3.3.5 JARs have been built with an openjdk
> version which requires IntBuffer implements an overridden method IntBuffer
> rewind().
> 
> java.lang.NoSuchMethodError: java.nio.IntBuffer.rewind()Ljava/nio/IntBuffer;
> 
> at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:341)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:308)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:257)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:202)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> 
>> On Tue, 28 Mar 2023 at 15:52, Viraj Jasani  wrote:
>> IIRC some of the ongoing major dependency upgrades (log4j 1 to 2, jersey 1
>> to 2 and junit 4 to 5) are blockers for java 11 compile + test stability.
>> On Tue, Mar 28, 2023 at 4:55 AM Steve Loughran > wrote:
>>> Now that hadoop 3.3.5 is out, i want to propose something new
>>> we switch branch-3.3 and trunk to being java11 only
>>> 1. java 11 has been out for years
>>> 2. oracle java 8 is no longer available under "premier support"; you
>>> can't really get upgrades
>>> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
>>> 3. openJDK 8 releases != oracle ones, and things you compile with them
>>> don't always link to oracle java 8 (some classes in java.nio have
>> added
>>> more overrides)
>>> 4. more and more libraries we want to upgrade to/bundle are java 11
>> only
>>> 5. moving to java 11 would cut our yetus build workload in half, and
>>> line up for adding java 17 builds instead.
>>> I know there are some outstanding issues still in
>>> https://issues.apache.org/jira/browse/HADOOP-16795 -but are they
>> blockers?
>>> Could we just move to java11 and enhance at our leisure, once java8 is no
>>> longer a concern.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Fwd: TAC supporting Berlin Buzzwords

[Ayush Saxena]

Forwarded a recieved.

-Ayush

-- Forwarded message -
From: Gavin McDonald 
Date: Fri, 24 Mar 2023 at 15:27
Subject: TAC supporting Berlin Buzzwords
To: 


PMCs,

Please forward to your dev and user lists.

Hi All,

The ASF Travel Assistance Committee is supporting taking up to six (6)
people
to attend Berlin Buzzwords In June this year.

This includes Conference passes, and travel & accommodation as needed.

Please see our website at https://tac.apache.org for more information and
how to apply.

Applications close on 15th April.

Good luck to those that apply.

Gavin McDonald (VP TAC)

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

+1(Binding)

* Built from source (x86 & ARM)
* Successful Native Build (x86 & ARM)
* Verified Checksums (x86 & ARM)
* Verified Signature (x86 & ARM)
* Checked the output of hadoop version (x86 & ARM)
* Verified the output of hadoop checknative (x86 & ARM)
* Ran some basic HDFS shell commands.
* Ran some basic Yarn shell commands.
* Played a bit with HDFS Erasure Coding.
* Ran TeraGen & TeraSort
* Browed through NN, DN, RM & NM UI
* Skimmed over the contents of website.
* Skimmed over the contents of maven repo.
* Selectively ran some HDFS & CloudStore tests

Thanx Steve for driving the release. Good Luck!!!

-Ayush

> On 20-Mar-2023, at 12:54 PM, Xiaoqiao He  wrote:
> 
> +1
> 
> * Verified signature and checksum of the source tarball.
> * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> * Run example mr applications and check the result - Pi & wordcount.
> * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
> 
> Thanks Steve for your work.
> 
> Best Regards,
> - He Xiaoqiao
> 
>> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki 
>> wrote:
>> 
>> +1
>> 
>> + verified the signature and checksum of the source tarball.
>> 
>> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
>> with native profile enabled.
>>   + launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>   + created encryption zone, put and read files via httpfs.
>>   + ran example MR wordcount over encryption zone.
>>   + checked the binary of container-executor.
>> 
>> + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux
>> 8 (aarch64).
>>   + ran smoke-tests of hdfs, yarn and mapreduce.
>> + built site documentation and skimmed the contents.
>>   +  Javadocs are contained.
>> 
>> Thanks,
>> Masatake Iwasaki
>> 
>>> On 2023/03/16 4:47, Steve Loughran wrote:
>>> Apache Hadoop 3.3.5
>>> 
>>> Mukund and I have put together a release candidate (RC3) for Hadoop
>> 3.3.5.
>>> 
>>> What we would like is for anyone who can to verify the tarballs,
>> especially
>>> anyone who can try the arm64 binaries as we want to include them too.
>>> 
>>> The RC is available at:
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>>> 
>>> The git tag is release-3.3.5-RC3, commit 706d88266ab
>>> 
>>> The maven artifacts are staged at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>>> 
>>> You can find my public key at:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> 
>>> Change log
>>> 
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>>> 
>>> Release notes
>>> 
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>>> 
>>> This is off branch-3.3 and is the first big release since 3.3.2.
>>> 
>>> Key changes include
>>> 
>>> * Big update of dependencies to try and keep those reports of
>>>   transitive CVEs under control -both genuine and false positives.
>>> * HDFS RBF enhancements
>>> * Critical fix to ABFS input stream prefetching for correct reading.
>>> * Vectored IO API for all FSDataInputStream implementations, with
>>>   high-performance versions for file:// and s3a:// filesystems.
>>>   file:// through java native io
>>>   s3a:// parallel GET requests.
>>> * This release includes Arm64 binaries. Please can anyone with
>>>   compatible systems validate these.
>>> * and compared to the previous RC, all the major changes are
>>>   HDFS issues.
>>> 
>>> Note, because the arm64 binaries are built separately on a different
>>> platform and JVM, their jar files may not match those of the x86
>>> release -and therefore the maven artifacts. I don't think this is
>>> an issue (the ASF actually releases source tarballs, the binaries are
>>> there for help only, though with the maven repo that's a bit blurred).
>>> 
>>> The only way to be consistent would actually untar the x86.tar.gz,
>>> overwrite its binaries with the arm stuff, retar, sign and push out
>>> for the vote. Even automating that would be risky.
>>> 
>>> Please try the release and vote. The vote will run for 5 days.
>>> 
>>> -Steve
>>> 
>> 
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> 
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

Count me in as well. I am almost done. So, you have 3 potential votes, can be 
happy now :) 
Thanx Steve for the efforts!!!

-Ayush

> On 19-Mar-2023, at 2:46 AM, Chris Nauroth  wrote:
> 
> Yes, I'm in progress on verification, so you can expect to get a vote from
> me. Thank you, Steve!
> 
> Chris Nauroth
> 
> 
>> On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta 
>> wrote:
>> Hi Steve
>> I will also do it by today/tomorrow.
>> Thanks,
>> Ashutosh
>> On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, > wrote:
>>> Thank you for this!
>>> Can anyone else with time do a review too? i really want to get this one
>>> done, now the HDFS issues are all resolved.
>>> I do not want this release to fall by the wayside through lack of votes
>>> alone. In fact, I would be very unhappy
 On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:
 +1 (non-binding)
 * Signature/Checksum: ok
 * Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
 * Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
 * Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
 Containerized deployments:
 * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper
>> 3.7
 * Deployed and started JHS, RM, NM
 * Hbase, hdfs CRUD looks good
 * Sample RowCount MapReduce job looks good
 * S3A tests with scale profile looks good
 On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
 
 wrote:
> Apache Hadoop 3.3.5
> Mukund and I have put together a release candidate (RC3) for Hadoop
 3.3.5.
> What we would like is for anyone who can to verify the tarballs,
 especially
> anyone who can try the arm64 binaries as we want to include them too.
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> The git tag is release-3.3.5-RC3, commit 706d88266ab
> The maven artifacts are staged at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> This is off branch-3.3 and is the first big release since 3.3.2.
> Key changes include
> * Big update of dependencies to try and keep those reports of
> transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
> high-performance versions for file:// and s3a:// filesystems.
> file:// through java native io
> s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
> compatible systems validate these.
> * and compared to the previous RC, all the major changes are
> HDFS issues.
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit
>> blurred).
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
> Please try the release and vote. The vote will run for 5 days.
> -Steve

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Ayush Saxena]

>
> I will highlight that I am completely fed up with doing this  release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.


hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.

Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.

Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.

So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's

-Ayush

On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:

> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> >issues in maven 3.9.0)
> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

>
>  And i
> think we need to change the PR template to mention transitive updates in
> the license bit too


Not sure if that is gonna help, People might ignore that or check that in
overconfidence. No harm though..

BTW Ozone has some cool stuff to handle this, it was added here:
https://github.com/apache/ozone/pull/2199

It checks for each PR, if the changes bring any new transitive dependency
or not and if it does, it flags that and then licence and all can be
managed. Worth exploring

-Ayush

On Sat, 25 Feb 2023 at 01:09, Steve Loughran 
wrote:

>  need this pr in too, https://github.com/apache/hadoop/pull/5429
>
>1. cuts back on some transitive dependencies from hadoop-aliyun
>2. fixes LICENSE-bin to be correct
>
> #2 is the blocker...and it looks like 3.2.x will also need fixup as well as
> the later ones -hadoop binaries have shipped without that file being up to
> date, but at least all the transitive stuff is correctly licensed. And i
> think we need to change the PR template to mention transitive updates in
> the license bit too
>
> if this goes in, I will do the rebuild on monday UK time
>
> On Thu, 23 Feb 2023 at 11:18, Steve Loughran  wrote:
>
> >
> > And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
> > maven releases (3.9.0)
> >
> > on a new local build with maven updated on homebrew (which i needed for
> > spark). so a code change too. That issue doesn't surface on our
> > release dockers, but will hit other people. especially over time. Going
> to
> > revert HADOOP-18590. Publish SBOM artifacts (#5281)
> >
> >
> >
> > On Thu, 23 Feb 2023 at 10:29, Steve Loughran 
> wrote:
> >
> >> ok, let me cancel, update those jiras and kick off again. that will save
> >> anyone else having to do their homework
> >>
> >> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
> >> wrote:
> >>
> >>> I'm now -1 as I found the wrong information on the top page (index.md).
> >>>
> >>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in
> HDFS
> >>> Router Based Federation.
> >>>
> >>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5
> before,
> >>> though it is actually not in branch-3.3. I corrected the fix version
> and
> >>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
> >>> Unfortunately, it won't be fixed soon. I should have let you know at
> that
> >>> time, sorry.  Supporting Observer NameNode in RBF is a prominent
> feature.
> >>> So I think we have to delete the description from the top page not to
> >>> confuse Hadoop users.
> >>>
> >>> - Takanobu
> >>>
> >>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
> >>>
> >>> > Thanks for driving the release, Steve and Mukund.
> >>> >
> >>> > I found that there were some jiras with wrong fix versions.
> >>> >
> >>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> >>> > - HDFS-16845
> >>> > - HADOOP-18345
> >>> >
> >>> > The fix versions didn't include 3.3.5, but actually, it is in
> 3.3.5-RC1
> >>> > (and it is not in release-3.3.4) :
> >>> > - HADOOP-17276
> >>> > - HDFS-13293
> >>> > - HDFS-15630
> >>> > - HDFS-16266
> >>> > - HADOOP-18003
> >>> > - HDFS-16310
> >>> > - HADOOP-18014
> >>> >
> >>> > I corrected all the wrong fix versions just now. I'm not sure we
> should
> >>> > revote it since it only affects the changelog.
> >>> >
> >>> > - Takanobu
> >>> >
> >>> > 2023年2月21日(火) 22:43 Steve Loughran :
> >>> >
> >>> >> Apache Hadoop 3.3.5
> >>> >>
> >>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
> >>> 3.3.5.
> >>> >>
> >>> >> What we would like is for anyone who can to verify the tarballs,
> >>> >> especially
> >>> >> anyone who can try the arm64 binaries as we want to include them
> too.
> >>> >>
> >>> >> The RC is available at:
> >>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
> >>> >>
> >>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
> >>> >>
> >>> >> The maven artifacts are staged at
> >>> >>
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
> >>> >>
> >>> >> You can find my public key at:
> >>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>> >>
> >>> >> Change log
> >>> >>
> >>> >>
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
> >>> >>
> >>> >> Release notes
> >>> >>
> >>> >>
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
> >>> >>
> >>> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>> >>
> >>> >> Key changes include
> >>> >>
> >>> >> * Big update of dependencies to try and keep those reports of
> >>> >>   transitive CVEs under control -both genuine and false positives.
> >>> >> * HDFS RBF enhancements
> >>> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >>> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>> >>   high-performance versions for file:// and s3a:// filesystems.
> >>> >>   

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

I haven't got a chance to deep dive into HADOOP-18324
<https://issues.apache.org/jira/browse/HADOOP-18324> which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
>From the PR uploaded on HDFS-16853
<https://issues.apache.org/jira/browse/HDFS-16853> it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.

++ @Owen O'Malley  is the original author of the
Hadoop Jira, maybe he can share some pointers about that.

-Ayush

On Thu, 5 Jan 2023 at 07:04, Chris Nauroth  wrote:

> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena  wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth  wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -Dtests

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?


I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]

if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>

Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.

Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know

Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]


[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484

-Ayush

On Tue, 3 Jan 2023 at 23:51, Chris Nauroth  wrote:

> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file " on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun  wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >  wrote:
> > >
> > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > >[INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > >[ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > >Please check the following and either correct the 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

Mostly or may be all of those failures are due to HADOOP-18324
, there is a Jira
tracking issues with TestLeaseRecovery2 linked to that as well HDFS-16853


-Ayush

On Wed, 28 Dec 2022 at 09:13, Viraj Jasani  wrote:

> -0 (non-binding)
>
> Output of hadoop-vote.sh:
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> Manual testing on local mini cluster:
> * Basic CRUD tests on Hdfs looks good
> * Sample MapReduce job looks good
> * S3A tests look good with scale profile (ITestS3AContractUnbuffer is
> flaky, but when run individually, it passes)
>
> Full build with all modules UT results for branch-3.3.5 latest HEAD are
> available on
>
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-3.3.5-java8-linux-x86_64/
>
> From the above build, there are some consistently failing tests, out of
> which only TestDataNodeRollingUpgrade passed locally, whereas rest of the
> tests are consistently failing locally as well, we might want to fix (or
> ignore, if required) them:
>
>
> org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshot#testSnapshotsOnErasureCodingDirAfterNNRestart
>
> org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart#testFileLengthWithHSyncAndClusterRestartWithOutDNsRegister
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart2
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryWithRenameAfterNameNodeRestart
>
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade#testWithLayoutChangeAndFinalize
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot#testSnapshotOpsOnRootReservedPath
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap#testReadRenamedSnapshotFileWithCheckpoint
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion#testApplyEditLogForDeletion
>
>
>
> On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran
> 
> wrote:
>
> > Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> >
> > Given the time of year it's a bit unrealistic to run a 5 day vote and
> > expect people to be able to test it thoroughly enough to make this the
> one
> > we can ship.
> >
> > What we would like is for anyone who can to verify the tarballs, and test
> > the binaries, especially anyone who can try the arm64 binaries. We've got
> > the building of those done and now the build file will incorporate them
> > into the release -but neither of us have actually tested it yet. Maybe I
> > should try it on my pi400 over xmas.
> >
> > The maven artifacts are up on the apache staging repo -they are the ones
> > from x86 build. Building and testing downstream apps will be incredibly
> > helpful.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >
> > The git tag is release-3.3.5-RC0, commit 3262495904d
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >   transitive CVEs under control -both genuine and false positive.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >   high-performance versions for file:// and s3a:// filesystems.
> >   file:// through java native io
> >   s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >   compatible systems validate these.
> >
> >
> > Please try the release and vote on it, even though i don't know what is a
> > good timeline here...i'm actually going on holiday in early jan. Mukund
> is
> > around and so can drive the process while I'm offline.
> >
> > Assuming we do have another iteration, the RC1 will not be before mid jan
> > for that reason
> >
> > Steve (and mukund)
> >
>

Re: exciting new content needed for the 3.3.5 index.md file

[Ayush Saxena]

Hi Steve,
I just went through the Jira's fixed in HDFS[1]
Sharing some stuff which I find good enough to find a mention.
1.HDFS-13522 , HDFS-16767
 & Related Jiras: Allow
Observer Reads in HDFS Router Based Federation.
2. HDFS-13248 : RBF
supports Client Locality
3. HDFS-16400 , HDFS-16399
, HDFS-16396
, HDFS-16397
, HDFS-16413
, HDFS-16457
: Makes a bunch of
Datanode level properties reconfigurable. The list is at[2], if required.

That is my small list from HDFS, I can come back before the RC if I find
something else as well, or If I figure out that I messed up with the above
stuff(I double checked the fix versions though...)

BTW. 2 unresolved Jira with FixVersion 3.3.5[3], do give a check on such
cases before you generate the RC

-Ayush

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HDFS%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20updated%20DESC
[2]
https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java#L346-L361
[3]
https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20updated%20DESC

On Fri, 2 Dec 2022 at 00:21, Steve Loughran 
wrote:

> The first "smoke test" RC is about to be up for people to play with, we are
> just testing things here and getting that arm build done.
>
> Can I have some content for the index.html page describing what has
> changed?
> hadoop-project/src/site/markdown/index.md.vm
>
> I can (and will) speak highly of stuff I've been involved in, but need
> contributions from others for what is new in this release in HDFS, YARN,
> and MR (other than the manifest committer).
>
> It'd be good to have a list of CVEs fixed by upgrading jars. Maybe we
> should have a transitive-CVE tag for all JIRAs which update a dependency
> for this, so that then we could have the release notes explicitly list
> these in their own section.
>
> Please submit changes to branch-3.3.5; use HADOOP-18470. as the jira for
> all the release notes.
>
>  thanks.
>

[jira] [Resolved] (YARN-11380) Fix hadoop-yarn-api module Java Doc Errors

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11380.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix hadoop-yarn-api module Java Doc Errors
> --
>
> Key: YARN-11380
> URL: https://issues.apache.org/jira/browse/YARN-11380
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When finishing YARN-11373, ps has java-doc compilation errors, I will fix all 
> java-doc compilation errors of hadoop-yarn-api module. I will pay attention 
> to the java doc compilation of jdk8 & jdk11 at the same time.
> {code:java}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.1:javadoc-no-fork 
> (default-cli) on project hadoop-yarn-api: An error has occurred in Javadoc 
> report generation: 
> [ERROR] Exit code: 1 - javadoc: warning - You have specified the HTML version 
> as HTML 4.01 by using the -html4 option.
> [ERROR] The default is currently HTML5 and the support for HTML 4.01 will be 
> removed
> [ERROR] in a future release. To suppress this warning, please ensure that any 
> HTML constructs
> [ERROR] in your comments are valid in HTML5, and remove the -html4 option.
> [ERROR] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5146/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java:97:
>  warning: no description for @throws
> [ERROR]* @throws YarnException
> [ERROR]  ^
> [ERROR] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5146/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java:98:
>  warning: no description for @throws
> [ERROR]* @throws IOException
> [ERROR]  ^
> [ERROR] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5146/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java:129:
>  warning: no description for @throws
> [ERROR]* @throws YarnException
> [ERROR]  ^
> [ERROR] 
> /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5146/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationBaseProtocol.java:130:
>  warning: no description for @throws
> [ERROR]* @throws IOException {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11381) Fix hadoop-yarn-common module Java Doc Errors

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11381.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix hadoop-yarn-common module Java Doc Errors
> -
>
> Key: YARN-11381
> URL: https://issues.apache.org/jira/browse/YARN-11381
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When finishing YARN-11373, ps  has java-doc compilation errors, I will fix 
> all java-doc compilation errors of hadoop-yarn-common module. I will pay 
> attention to the java doc compilation of jdk8 & jdk11 at the same time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] JIRA Public Signup Disabled

[Ayush Saxena]

Thanx Mingliang Liu, Considering the traffic around private@ for new
accounts(surprisingly), I went ahead and updated our Contribution
Guidelines[1], If anyone feels I made a mistake in the content or it can
rephrased in a better way, feel free to change it or let me know.

In future if the traffic increases with such requests, we can obviously
come back here and discuss the 2nd option or any other possible options.

Thanx Everyone!!!

-Ayush

[1]
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-RequestingforaJiraaccount

On Fri, 25 Nov 2022 at 06:16, Mingliang Liu  wrote:

> Thanks Ayush for taking care of this. I think option 1 sounds good.
>
> > On Nov 22, 2022, at 2:51 PM, Ayush Saxena  wrote:
> >
> > Hi Folks,
> > Just driving the attention towards the recent change from Infra, which
> > disables new people from creating a Jira account, in order to prevent
> spams
> > around JIRA.
> > So, the new Jira account creation request needs to be routed via the PMC
> of
> > the project.
> > So, we have 2 options, which I can think of:
> > 1. Update the contribution guidelines to route such requests to private@
> > 2. Create a dedicated ML for it. A couple of projects which I know did
> that.
> >
> > The Infra page: https://infra.apache.org/jira-guidelines.html
> >
> > Let me know what folks think, if nothing, I will go with the 1st option
> and
> > update the contribution guidelines mostly by next week or a week after
> that.
> >
> > -Ayuhs
>
>

[DISCUSS] JIRA Public Signup Disabled

[Ayush Saxena]

Hi Folks,
Just driving the attention towards the recent change from Infra, which
disables new people from creating a Jira account, in order to prevent spams
around JIRA.
So, the new Jira account creation request needs to be routed via the PMC of
the project.
So, we have 2 options, which I can think of:
1. Update the contribution guidelines to route such requests to private@
2. Create a dedicated ML for it. A couple of projects which I know did that.

The Infra page: https://infra.apache.org/jira-guidelines.html

Let me know what folks think, if nothing, I will go with the 1st option and
update the contribution guidelines mostly by next week or a week after that.

-Ayuhs

[jira] [Resolved] (YARN-11334) [Federation] Improve SubClusterState#fromString parameter and LogMessage

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11334.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> [Federation] Improve SubClusterState#fromString parameter and LogMessage
> 
>
> Key: YARN-11334
> URL: https://issues.apache.org/jira/browse/YARN-11334
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Trivial
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The SubClusterState object has a fromString method, which can be improved in 
> the following 2 places
> {code:java}
> /**
>  * Convert a string into {@code SubClusterState}.
>  *
>  * @param x the string to convert in SubClusterState
>  * @return the respective {@code SubClusterState}
>  */
> public static SubClusterState fromString(String x) {
>   try {
> return SubClusterState.valueOf(x);
>   } catch (Exception e) {
> LOG.error("Invalid SubCluster State value in the StateStore does not"
> + " match with the YARN Federation standard.");
> return null;
>   }
> } {code}
>  * The parameter is named x, which cannot well express the meaning of the 
> input parameter.
>  * The error log does not print the error value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11240) Fix incorrect placeholder in yarn-module

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11240.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix incorrect placeholder in yarn-module
> 
>
> Key: YARN-11240
> URL: https://issues.apache.org/jira/browse/YARN-11240
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Try to deal with the moudle problem at a time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11305) Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241(#4703)

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11305.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to trunk.
Thanx [~slfan1989] for the contribution!!!

> Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After 
> YARN-11241(#4703)
> -
>
> Key: YARN-11305
> URL: https://issues.apache.org/jira/browse/YARN-11305
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Yarn's unit tests, with parallel mode enabled
> {code:java}
> /usr/bin/mvn --batch-mode 
> -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4846/yetus-m2/hadoop-trunk-patch-0
>  -Ptest-patch -Dsurefire.rerunFailingTestsCount=2 -Pparallel-tests 
> -P!shelltest -Pnative -Drequire.fuse -Drequire.openssl -Drequire.snappy 
> -Drequire.valgrind -Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui clean 
> test -fae{code}
> Because the automatically generated application_id is the same, 
> *testLocalFileDeletionAfterUpload* is affected by 
> *testLocalFileRemainsAfterUploadOnCleanupDisable*
> testLocalFileDeletionAfterUpload needs to check that the log of an 
> application is deleted, and testLocalFileRemainsAfterUploadOnCleanupDisable 
> regenerates the log of the same application.
> Error Message
> {code:java}
> Directory 
> [/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4846/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestLogAggregationService-localLogDir/application_1234_0001]
>  was not deleted {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11286) Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter configurable

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11286.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter 
> configurable
> -
>
> Key: YARN-11286
> URL: https://issues.apache.org/jira/browse/YARN-11286
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> AsyncDispatcher#printEventDetailsExecutor thread pool parameters are 
> hard-coded, extract this part of hard-coded configuration parameters to the 
> configuration file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11274) Improve Nodemanager#NodeStatusUpdaterImpl Log

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11274.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Improve Nodemanager#NodeStatusUpdaterImpl Log
> -
>
> Key: YARN-11274
> URL: https://issues.apache.org/jira/browse/YARN-11274
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> 1.When RM performs master-slave switching, NM will re-register, report the 
> list of running apps, and print the number of running apps
> 2.registerWithRM#containerReports is enough to print the number of prints, 
> because the above calling process has already printed the container list, so 
> there is no need to repeat printing
> 3.Use placeholders to display logs to avoid string concatenation



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

2022 ASF Community Survey

[Ayush Saxena]


Hello everyone,

The 2022 ASF Community Survey is looking to gather scientific data that
allows us to understand our community better, both in its demographic
composition, and also in collaboration styles and preferences. We want to
find areas where we can continue to do great work, and others where we need
to provide more support so that our projects can keep growing healthy and
diverse.

If you have an apache.org email, you should have received an email with an
invitation to take the 2022 ASF Community Survey. Please take 15 minutes to
complete it.

If you do not have an apache.org email address or you didn’t receive a
link, please follow this link to the survey:

https://edi-asf.limesurvey.net/912832?lang=en

You can find information about privacy on the survey’s Confluence page.
<https://cwiki.apache.org/confluence/display/EDI/Survey+-+Launch+Plan> The
last surveys of this kind were implemented in 2016 and 2020, which means we
are finally in a position to see trends over time.

Your participation is paramount to the success of this project! Please
consider filling out the survey, and share this news with your fellow
Apache contributors. As individuals form the Apache community, your opinion
matters: we want to hear your voice.

If you have any questions about the survey or otherwise, please reach out
to us!

Kindly,

Ayush Saxena
On Behalf of:

Katia Rojas

V.P. of Diversity and Inclusion

The Apache Software Foundation

[jira] [Resolved] (YARN-11203) Fix typo in hadoop-yarn-server-router module

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11203.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix typo in hadoop-yarn-server-router module
> 
>
> Key: YARN-11203
> URL: https://issues.apache.org/jira/browse/YARN-11203
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: federation
>Affects Versions: 3.4.0, 3.3.4
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Fix typo in hadoop-yarn-server-router module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0

[Ayush Saxena]

+1(Binding)

* Built from Source
* Successful native build on Ubuntu 18.04
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran Basic HDFS shell commands
* Ran basic MR example Jobs (TeraGen/TeraSort & TeraValidate)
* Browsed through UI(NN, DN, RM, NM & JHS)
* Skimmed through the contents of ChangeLog & ReleaseNotes. Look Good

Thanx Masatake for driving the release, Good Luck!!!

-Ayush

On Thu, 21 Jul 2022 at 23:28, Chris Nauroth  wrote:

> I'm changing my vote to +1 (binding).
>
> Masatake and Ashutosh, thank you for investigating.
>
> I reran tests without the parallel options, and that mostly addressed the
> failures. Maybe the tests in question are just not sufficiently isolated to
> support parallel execution. That looks to be the case for TestFsck, where
> the failure was caused by missing audit log entries. This test works by
> toggling global logging state, so I can see why multi-threaded execution
> might confuse the test.
>
> Chris Nauroth
>
>
> On Thu, Jul 21, 2022 at 12:01 AM Ashutosh Gupta <
> ashutoshgupta...@gmail.com>
> wrote:
>
> > +1(non-binding)
> >
> > * Builds from source look good.
> > * Checksums and signatures are correct.
> > * Running basic HDFS and MapReduce commands looks good.
> >
> > > * TestAMRMProxy - Not able to reproduce in local
> > > * TestFsck - I can see failure only I can see is
> >  TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
> > HDFS-15038
> > > * TestSLSStreamAMSynth - Not able to reproduce in local
> > > * TestServiceAM - Not able to reproduce in local
> >
> > Thanks Masatake for driving this release.
> >
> > On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki <
> > iwasak...@oss.nttdata.com>
> > wrote:
> >
> > > Hi developers,
> > >
> > > I'm still waiting for your vote.
> > > I'm considering the intermittent test failures mentioned by Chris are
> not
> > > blocker.
> > > Please file a JIRA and let me know if you find a blocker issue.
> > >
> > > I will appreciate your help for the release process.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > > On 2022/07/20 14:50, Masatake Iwasaki wrote:
> > > >> TestServiceAM
> > > >
> > > > I can see the reported failure of TestServiceAM in some "Apache
> Hadoop
> > > qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > > > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> > > GenericTestUtils#waitFor for stabilizing the
> > > testContainersReleasedWhenPreLaunchFails.
> > > > YARN 8867 did not modified other code under hadoop-yarn-services.
> > > > If it is the case, TestServiceAM can be tagged as flaky in
> branch-3.2.
> > > >
> > > >
> > > > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> > > >> Thanks for testing the RC0, Chris.
> > > >>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>
> > > >> I could not reproduce the test failures on my local.
> > > >>
> > > >> For TestFsck, if the failed test case is
> > > testFsckListCorruptSnapshotFiles,
> > > >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> > > >>
> > > >> The failure of TestSLSStreamAMSynth looks frequently reported by
> > > >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> > > >> It could be tagged as known flaky test.
> > > >>
> > > >> On 2022/07/20 9:15, Chris Nauroth wrote:
> > > >>> -0 (binding)
> > > >>>
> > > >>> * Verified all checksums.
> > > >>> * Verified all signatures.
> > > >>> * Built from source, including native code on Linux.
> > > >>>  * mvn clean package -Pnative -Psrc -Drequire.openssl
> > > -Drequire.snappy
> > > >>> -Drequire.zstd -DskipTests
> > > >>> * Tests mostly passed, but see below.
> > > >>>  * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > >>>
> > > >>> The following are new test failures for me on 3.2.4:
> > > >>> * TestAMRMProxy
> > > >>> * TestFsck
> > > >>> * TestSLSStreamAMSynth
> > > >>> * TestServiceAM
> > > >>>
> > > >>> The following tests also failed, but they also fail for me on
> 3.2.3,
> > so
> > > >>> they aren't likely to be related to this release candidate:
> > > >>> * TestCapacitySchedulerNodeLabelUpdate
> > > >>> * TestFrameworkUploader
> > > >>> * TestSLSGenericSynth
> > > >>> * TestSLSRunner
> > > >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> > > >>>
> > > >>> I'm not voting a full -1, because I haven't done any root cause
> > > analysis on
> > > >>> these new test failures. I don't know if it's a quirk to my
> > > environment,
> > > >>> though I'm using the start-build-env.sh Docker container, so any
> > build
> > > >>> dependencies should be consistent. I'd be comfortable moving ahead
> if
> > > >>> others are seeing these tests pass.
> > > >>>
> > > >>> Chris Nauroth
> > > >>>
> > > >>>
> > > >>> On Thu, Jul 14, 2022 at 7:57 AM 

[jira] [Resolved] (YARN-11192) TestRouterWebServicesREST failing after YARN-9827

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11192.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> TestRouterWebServicesREST failing after YARN-9827
> -
>
> Key: YARN-11192
> URL: https://issues.apache.org/jira/browse/YARN-11192
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In YARN-9827, the following modifications:
> {code:java}
> GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of 
> connection and service unavailable exception instead of 
> INTERNAL_SERVICE_ERROR. {code}
> This modification caused all of YARN Federation's TestRouterWebServicesREST 
> unit tests to fail
> {code:java}
> [ERROR] Tests run: 201, Failures: 15, Errors: 0, Skipped: 0, Flakes: 2
> .
> [ERROR] 
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST.testUpdateAppStateXML(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServicesREST)
> [ERROR]   Run 1: TestRouterWebServicesREST.testUpdateAppStateXML:774 
> expected:<500> but was:<503>
> [ERROR]   Run 2: TestRouterWebServicesREST.testUpdateAppStateXML:774 
> expected:<500> but was:<503>
> [ERROR]   Run 3: TestRouterWebServicesREST.testUpdateAppStateXML:774 
> expected:<500> but was:<503> {code}
> Report-URL:
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4464/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-11142) Remove unused Imports in Hadoop YARN project

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-11142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved YARN-11142.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Remove unused Imports in Hadoop YARN project
> 
>
> Key: YARN-11142
> URL: https://issues.apache.org/jira/browse/YARN-11142
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> h3. Optimize Imports to keep code clean
>  # Remove any unused imports



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Ayush Saxena]

+1,
* Built from source
* Verified Checksums
* Verified Signatures
* Successful RAT check
* Ran some basic HDFS shell commands.
* Browsed through the UI(NN,DN,RM,NM,JHS)

Thanx Masatake for driving the release, Good Luck!!!

-Ayush

On Mon, 30 May 2022 at 03:14, Chris Nauroth  wrote:

> +1 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Almost all unit tests passed.
> * mvn clean test -Pnative -Dparallel-tests -Drequire.snappy
> -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3
> -DtestsThreadCount=8
> * TestBookKeeperHACheckpoints consistently has a few failures.
> * TestCapacitySchedulerNodeLabelUpdate is flaky, intermittently timing
> out.
>
> These test failures don't look significant enough to hold up a release, so
> I'm still voting +1.
>
> Chris Nauroth
>
>
> On Sun, May 29, 2022 at 2:35 AM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
>> Thanks for the help, Ayush.
>>
>> I committed HADOOP-16663/HADOOP-16664 and cherry-picked HADOOP-16985 to
>> branch-2.10 (and branch-3.2).
>> If I need to cut RC1, I will try cherry-picking them to branch-2.10.2
>>
>> Masatake Iwasaki
>>
>>
>> On 2022/05/28 5:23, Ayush Saxena wrote:
>> > The checksum stuff was addressed in HADOOP-16985, so that filename
>> stuff is
>> > sorted only post 3.3.x
>> > BTW it is a known issue:
>> >
>> https://issues.apache.org/jira/browse/HADOOP-16494?focusedCommentId=16927236=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16927236
>> >
>> > Must not be a blocker for us
>> >
>> > The RAT check failing with dependency issue. That also should work post
>> > 3.3.x because there is no Hadoop-maven-plugin dependency in
>> Hadoop-yarn-api
>> > module post 3.3.x, HADOOP-16560 removed it.
>> > Ref:
>> >
>> https://github.com/apache/hadoop/pull/1496/files#diff-f5d219eaf211871f9527ae48da59586e7e9958ea7649de74a1393e599caa6dd6L121-R122
>> >
>> > So, that is why the RAT check passes for 3.3.x+ without the need of this
>> > module. Committing HADOOP-16663, should solve this though.(I haven't
>> tried
>> > though, just by looking at the problem)
>> >
>> > Good to have patches, but doesn't look like blockers to me. kind of
>> build
>> > related stuffs only, nothing bad with our core Hadoop code.
>> >
>> > -Ayush
>> >
>> > On Sat, 28 May 2022 at 01:04, Viraj Jasani  wrote:
>> >
>> >> +0 (non-binding),
>> >>
>> >> * Signature/Checksum looks good, though I am not sure where
>> >> "target/artifacts" is coming from for the tars, here is the diff (this
>> was
>> >> the case for 2.10.1 as well but checksum was correct):
>> >>
>> >> 1c1
>> >> < SHA512 (hadoop-2.10.2-site.tar.gz) =
>> >>
>> >>
>> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
>> >> ---
>> >>> SHA512 (target/artifacts/hadoop-2.10.2-site.tar.gz) =
>> >>
>> >>
>> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
>> >> 1c1
>> >> < SHA512 (hadoop-2.10.2-src.tar.gz) =
>> >>
>> >>
>> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
>> >> ---
>> >>> SHA512 (target/artifacts/hadoop-2.10.2-src.tar.gz) =
>> >>
>> >>
>> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
>> >> 1c1
>> >> < SHA512 (hadoop-2.10.2.tar.gz) =
>> >>
>> >>
>> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
>> >> ---
>> >>> SHA512 (target/artifacts/hadoop-2.10.2.tar.gz) =
>> >>
>> >>
>> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
>> >>
>> >> However, checksums are correct.
>> >>
>> >> * Builds from source look good
>> >>   - mvn clean 

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Ayush Saxena]

The checksum stuff was addressed in HADOOP-16985, so that filename stuff is
sorted only post 3.3.x
BTW it is a known issue:
https://issues.apache.org/jira/browse/HADOOP-16494?focusedCommentId=16927236=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16927236

Must not be a blocker for us

The RAT check failing with dependency issue. That also should work post
3.3.x because there is no Hadoop-maven-plugin dependency in Hadoop-yarn-api
module post 3.3.x, HADOOP-16560 removed it.
Ref:
https://github.com/apache/hadoop/pull/1496/files#diff-f5d219eaf211871f9527ae48da59586e7e9958ea7649de74a1393e599caa6dd6L121-R122

So, that is why the RAT check passes for 3.3.x+ without the need of this
module. Committing HADOOP-16663, should solve this though.(I haven't tried
though, just by looking at the problem)

Good to have patches, but doesn't look like blockers to me. kind of build
related stuffs only, nothing bad with our core Hadoop code.

-Ayush

On Sat, 28 May 2022 at 01:04, Viraj Jasani  wrote:

> +0 (non-binding),
>
> * Signature/Checksum looks good, though I am not sure where
> "target/artifacts" is coming from for the tars, here is the diff (this was
> the case for 2.10.1 as well but checksum was correct):
>
> 1c1
> < SHA512 (hadoop-2.10.2-site.tar.gz) =
>
> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
> ---
> > SHA512 (target/artifacts/hadoop-2.10.2-site.tar.gz) =
>
> 3055a830003f5012660d92da68a317e15da5b73301c2c73cf618e724c67b7d830551b16928e0c28c10b66f04567e4b6f0b564647015bacc4677e232c0011537f
> 1c1
> < SHA512 (hadoop-2.10.2-src.tar.gz) =
>
> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
> ---
> > SHA512 (target/artifacts/hadoop-2.10.2-src.tar.gz) =
>
> 483b6a4efd44234153e21ffb63a9f551530a1627f983a8837c655ce1b8ef13486d7178a7917ed3f35525c338e7df9b23404f4a1b0db186c49880448988b88600
> 1c1
> < SHA512 (hadoop-2.10.2.tar.gz) =
>
> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
> ---
> > SHA512 (target/artifacts/hadoop-2.10.2.tar.gz) =
>
> 13e95907073d815e3f86cdcc24193bb5eec0374239c79151923561e863326988c7f32a05fb7a1e5bc962728deb417f546364c2149541d6234221b00459154576
>
> However, checksums are correct.
>
> * Builds from source look good
>  - mvn clean install  -DskipTests
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Rat check, if run before building from source locally, fails with error:
>
> [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:2.10.2 or one of its
> dependencies could not be resolved: Could not find artifact
> org.apache.hadoop:hadoop-maven-plugins:jar:2.10.2 in central (
> https://repo.maven.apache.org/maven2) -> [Help 1]
> [ERROR]
>
> However, once we build locally, rat check passes (because
> hadoop-maven-plugins 2.10.2 would be present in local .m2).
> Also, hadoop-maven-plugins:2.10.2 is available here
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1350/org/apache/hadoop/hadoop-maven-plugins/2.10.2/
>
> * Ran sample HDFS and MapReduce commands, look good.
>
> Until we release Hadoop artifacts, hadoop-maven-plugins for that release
> would not be present in the central maven repository, hence I am still
> wondering how rat check failed only for this RC and not for any of previous
> release RCs. hadoop-vote.sh always runs rat check before building from
> source locally.
>
>
> On Tue, May 24, 2022 at 7:41 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
> > Hi all,
> >
> > Here's Hadoop 2.10.2 release candidate #0:
> >
> > The RC is available at:
> >https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
> >
> > The RC tag is at:
> >https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1350
> >
> > You can find my public key at:
> >https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Ayush Saxena]

Have raised HADOOP-18241
There is a WIP PR linked to it as well

https://issues.apache.org/jira/browse/HADOOP-18241

On Wed, 11 May 2022 at 01:28, Ayush Saxena  wrote:

> +1 for moving to Java 11 for trunk.
> If nobody objects by EOW will create a jira for that
>
> -Ayush
>
> On 10-May-2022, at 11:18 PM, Gautham Banasandra 
> wrote:
>
> 
> +1 for moving to Java 11.
>
> Thanks,
> --Gautham
>
> On Mon, 9 May 2022 at 19:50, Steve Vaughan  wrote:
>
>> +1 on Java 11 for branch-3.3
>>
>> I'd also like to start addressing the test instability by addressing
>> collisions between tests.  Currently the tests are working in the same
>> shared resources either directly through the system properties like
>> test.build.data, indirectly through utility classes like GenericTestUtils,
>> or modifying shared resources within target/test-classes.  I've been
>> working towards cleaning up flaky tests, and I plan on submitting some
>> patches today.
>> ------
>> *From:* Steve Loughran 
>> *Sent:* Monday, May 9, 2022 7:50 AM
>> *To:* Ayush Saxena 
>> *Cc:* Gautham Banasandra ; Hadoop Common <
>> common-...@hadoop.apache.org>; Hdfs-dev ;
>> yarn-dev ; d...@hadoop.apache.org <
>> d...@hadoop.apache.org>
>> *Subject:* Re: [DISCUSS] Enabling all platform builds in CI for all
>> Hadoop PRs
>>
>> how about for trunk we -wait for it- declare it java11 only?
>>
>> do that and we cut out a lot of the CI build.
>>
>> we would have to mandate a test run through branch-3.3 before any
>> cherrypick there
>>
>> On Sat, 7 May 2022 at 11:19, Ayush Saxena  wrote:
>>
>> > Three for trunk:
>> > Java-8
>> >
>> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/
>> >
>> > Java-11
>> >
>> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/
>> >
>> > ARM
>> >
>> >
>> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/
>> >
>> > -Ayush
>> >
>> > >
>> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra 
>> > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > Although validating cross platform builds at pre-commit
>> > > would be the most effective approach, I understand the
>> > > huge disadvantage caused by the slowdown. The best
>> > > way to tackle this would be to enable parallel builds for
>> > > the different platforms. I had given it a shot about a year
>> > > ago[1], it didn't go well and ran into all sorts of random
>> > > errors. I think we should make the parallel builds run on
>> > > different agents as opposed to starting the builds parallelly
>> > > on the same agent (which is what I had done earlier).
>> > >
>> > > So, I'll settle down to integrating the full suite of platform
>> > > builds into the nightly builds. Could anyone please point
>> > > me to the Jenkins job for this?
>> > >
>> > > [1] = https://github.com/apache/hadoop/pull/3166
>> > >
>> > > Thanks,
>> > > --Gautham
>> > >
>> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena 
>> wrote:
>> > >>
>> > >> From functional point of view it does makes sense to validate all the
>> > >> platforms as part of the builds, but the Pre commits builds taking
>> time
>> > is
>> > >> now no longer a small things, In past one year or may be two, we have
>> > >> already increased it more than twice as compared to what it was
>> before,
>> > If
>> > >> someone has a change in HDFS, which includes both hdfs-client &
>> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2
>> > hours.
>> > >> With the current state, I don't think we should just go and add these
>> > >> extra overheads. Having them as part of the nightly builds does makes
>> > sense
>> > >> for now.
>> > >>
>> > >> In future if we feel there is a strong need for this and we start to
>> see
>> > >> very frequent failures in some other platforms and we are left with
>> no
>> > >> other option but to integrate it in our pre-commit jobs, we can
>> explore
>> > >> having these build phases running in parallel, along with trying
>> other
>> > >&

Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Ayush Saxena]

+1,
* Built from source.
* Successful native build on Ubuntu 18.04
* Verified Checksums.
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Verified Signature.
* Successful RAT check
* Ran basic HDFS shell commands.
* Ran basic YARN shell commands.
* Verified version in hadoop version command and UI
* Ran some MR example Jobs.
* Browsed UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer)
* Browsed the contents of Maven Artifacts.
* Browsed the contents of the website.

Thanx Steve for driving the release, Good Luck!!!

-Ayush

On Mon, 16 May 2022 at 08:20, Xiaoqiao He  wrote:

> +1(binding)
>
> * Verified signature and checksum of the source tarball.
> * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/mv/rm and check the result.
> * Run example mr applications and check the result - Pi & wordcount.
> * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
>
> Thanks Steve for your work.
>
> - He Xiaoqiao
>
> On Mon, May 16, 2022 at 4:25 AM Viraj Jasani  wrote:
> >
> > +1 (non-binding)
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_301): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_301): ok
> >  - mvn clean install  -DskipTests
> > * Built tar from source (1.8.0_301): ok
> >  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >
> > HDFS, MapReduce and HBase (2.5) CRUD functional testing on
> > pseudo-distributed mode looks good.
> >
> >
> > On Wed, May 11, 2022 at 10:26 AM Steve Loughran
> 
> > wrote:
> >
> > > I have put together a release candidate (RC1) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/
> > >
> > > The git tag is release-3.3.3-RC1, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1349/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > > * The critical fixes which shipped in the 3.2.3 release.
> > > * CVEs in our code and dependencies
> > > * Shaded client packaging issues.
> > > * A switch from log4j to reload4j
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> > > which contain CVEs removed. Even though hadoop never used those
> classes,
> > > they regularly raised alerts on security scans and concen from users.
> > > Switching to the forked project allows us to ship a secure logging
> > > framework. It will complicate the builds of downstream
> > > maven/ivy/gradle projects which exclude our log4j artifacts, as they
> > > need to cut the new dependency instead/as well.
> > >
> > > See the release notes for details.
> > >
> > > This is the second release attempt. It is the same git commit as
> before,
> > > but
> > > fully recompiled with another republish to maven staging, which has bee
> > > verified by building spark, as well as a minimal test project.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Ayush Saxena]

+1 for moving to Java 11 for trunk.
If nobody objects by EOW will create a jira for that

-Ayush

> On 10-May-2022, at 11:18 PM, Gautham Banasandra  wrote:
> 
> 
> +1 for moving to Java 11.
> 
> Thanks,
> --Gautham
> 
>> On Mon, 9 May 2022 at 19:50, Steve Vaughan  wrote:
>> +1 on Java 11 for branch-3.3
>> 
>> I'd also like to start addressing the test instability by addressing 
>> collisions between tests.  Currently the tests are working in the same 
>> shared resources either directly through the system properties like 
>> test.build.data, indirectly through utility classes like GenericTestUtils, 
>> or modifying shared resources within target/test-classes.  I've been working 
>> towards cleaning up flaky tests, and I plan on submitting some patches today.
>> From: Steve Loughran 
>> Sent: Monday, May 9, 2022 7:50 AM
>> To: Ayush Saxena 
>> Cc: Gautham Banasandra ; Hadoop Common 
>> ; Hdfs-dev ; 
>> yarn-dev ; d...@hadoop.apache.org 
>> 
>> Subject: Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs
>>  
>> how about for trunk we -wait for it- declare it java11 only?
>> 
>> do that and we cut out a lot of the CI build.
>> 
>> we would have to mandate a test run through branch-3.3 before any
>> cherrypick there
>> 
>> On Sat, 7 May 2022 at 11:19, Ayush Saxena  wrote:
>> 
>> > Three for trunk:
>> > Java-8
>> >
>> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/
>> >
>> > Java-11
>> >
>> > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/
>> >
>> > ARM
>> >
>> > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/
>> >
>> > -Ayush
>> >
>> > >
>> > > On 07-May-2022, at 11:49 AM, Gautham Banasandra 
>> > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > Although validating cross platform builds at pre-commit
>> > > would be the most effective approach, I understand the
>> > > huge disadvantage caused by the slowdown. The best
>> > > way to tackle this would be to enable parallel builds for
>> > > the different platforms. I had given it a shot about a year
>> > > ago[1], it didn't go well and ran into all sorts of random
>> > > errors. I think we should make the parallel builds run on
>> > > different agents as opposed to starting the builds parallelly
>> > > on the same agent (which is what I had done earlier).
>> > >
>> > > So, I'll settle down to integrating the full suite of platform
>> > > builds into the nightly builds. Could anyone please point
>> > > me to the Jenkins job for this?
>> > >
>> > > [1] = https://github.com/apache/hadoop/pull/3166
>> > >
>> > > Thanks,
>> > > --Gautham
>> > >
>> > >> On Fri, 6 May 2022 at 21:04, Ayush Saxena  wrote:
>> > >>
>> > >> From functional point of view it does makes sense to validate all the
>> > >> platforms as part of the builds, but the Pre commits builds taking time
>> > is
>> > >> now no longer a small things, In past one year or may be two, we have
>> > >> already increased it more than twice as compared to what it was before,
>> > If
>> > >> someone has a change in HDFS, which includes both hdfs-client &
>> > >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2
>> > hours.
>> > >> With the current state, I don't think we should just go and add these
>> > >> extra overheads. Having them as part of the nightly builds does makes
>> > sense
>> > >> for now.
>> > >>
>> > >> In future if we feel there is a strong need for this and we start to see
>> > >> very frequent failures in some other platforms and we are left with no
>> > >> other option but to integrate it in our pre-commit jobs, we can explore
>> > >> having these build phases running in parallel, along with trying other
>> > >> phases also to run in parallel like compilation/javadoc builds of JDK-8
>> > &
>> > >> JDK-11 can run in parallel and may be explore other opportunities as
>> > well
>> > >> to compensate for this time.
>> > >>
>> > >> For now lets just integrate it our nightly builds only and circle back
>> > >&

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Ayush Saxena]

Three for trunk:
Java-8

https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/

Java-11

https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/

ARM

https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/

-Ayush

> 
> On 07-May-2022, at 11:49 AM, Gautham Banasandra  wrote:
> 
> Hi all,
> 
> Although validating cross platform builds at pre-commit
> would be the most effective approach, I understand the
> huge disadvantage caused by the slowdown. The best
> way to tackle this would be to enable parallel builds for
> the different platforms. I had given it a shot about a year
> ago[1], it didn't go well and ran into all sorts of random
> errors. I think we should make the parallel builds run on
> different agents as opposed to starting the builds parallelly
> on the same agent (which is what I had done earlier).
> 
> So, I'll settle down to integrating the full suite of platform
> builds into the nightly builds. Could anyone please point
> me to the Jenkins job for this?
> 
> [1] = https://github.com/apache/hadoop/pull/3166
> 
> Thanks,
> --Gautham
> 
>> On Fri, 6 May 2022 at 21:04, Ayush Saxena  wrote:
>> 
>> From functional point of view it does makes sense to validate all the
>> platforms as part of the builds, but the Pre commits builds taking time is
>> now no longer a small things, In past one year or may be two, we have
>> already increased it more than twice as compared to what it was before, If
>> someone has a change in HDFS, which includes both hdfs-client &
>> hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours.
>> With the current state, I don't think we should just go and add these
>> extra overheads. Having them as part of the nightly builds does makes sense
>> for now.
>> 
>> In future if we feel there is a strong need for this and we start to see
>> very frequent failures in some other platforms and we are left with no
>> other option but to integrate it in our pre-commit jobs, we can explore
>> having these build phases running in parallel, along with trying other
>> phases also to run in parallel like compilation/javadoc builds of JDK-8 &
>> JDK-11 can run in parallel and may be explore other opportunities as well
>> to compensate for this time.
>> 
>> For now lets just integrate it our nightly builds only and circle back
>> again here in future if the need be.
>> 
>> -Ayush
>> 
>>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang  wrote:
>>> 
>>> Running builds for all platforms for each and every PR seems too
>>> excessive.
>>> 
>>> How about doing all platform builds in the nightly jobs?
>>> 
>>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran >>> 
>>> wrote:
>>> 
>>>> I'm not enthusiastic here as it not only makes the builds slower, it
>>>> reduces the #of builds we can through a day
>>>> 
>>>> one thing I am wondering is could we remove java8 support on some
>>> branches?
>>>> 
>>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and
>>> this
>>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11
>>>> only.
>>>> that would cut buld and test time for those trunk PRs in half...after
>>> which
>>>> the preospect of building on more than one platform becomes more viable.
>>>> 
>>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra 
>>>> wrote:
>>>> 
>>>>> Hi Hadoop devs,
>>>>> 
>>>>> Last week, there was a Hadoop build failure on Debian 10 caused by
>>>>> https://github.com/apache/hadoop/pull/3988. In
>>> dev-support/jenkins.sh,
>>>>> there's the capability to build and test Hadoop across the supported
>>>>> platforms. Currently, we're limiting this only for those PRs having
>>> only
>>>>> C/C++ changes[1], since C/C++ changes are more likely to cause
>>>>> cross-platform build issues and bypassing the full platform build for
>>> non
>>>>> C/C++ PRs would save a great deal of CI time. However, the build
>>> failure
>>>>> caused by PR #3988 motivates me to enable the capability to build and
>>>>> test Hadoop for all the supported platforms for ALL the PRs.
>>>>> 
>>>>> While this may cause longer CI run duration for each PR, it would
>>>>> immensely minimize the risk of breaking Hadoop across platforms and
>>>>> saves us a lot of debugging time. Kindly post your opinion regarding
>>> this
>>>>> and I'll move to enable this capability for all PRs if the response is
>>>>> sufficiently positive.
>>>>> 
>>>>> [1] =
>>>>> 
>>>>> 
>>>> 
>>> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> --Gautham
>>>>> 
>>>> 
>>> 
>> 

Re: [VOTE] Release Apache Hadoop 3.3.3

[Ayush Saxena]

Hmm, I see the artifacts ideally should have got overwritten by the new RC, but 
they didn’t. The reason seems like the staging path shared doesn’t have any 
jars…
That is why it was picking the old jars. I think Steve needs to run mvn deploy 
again…

Sent from my iPhone

> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> 
> 
>> 
>> Chao can you use the one that Steve mentioned in the mail?
> 
> Hmm how do I do that? Typically after closing the RC in nexus the
> release bits will show up in
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> and Spark build will be able to pick them up for testing. However in
> this case I don't see any 3.3.3 jars in the URL.
> 
>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena  wrote:
>> 
>> There were two 3.3.3 staged. The earlier one was with skipShade, the date 
>> was also april 22, I archived that. Chao can you use the one that Steve 
>> mentioned in the mail?
>> 
>>> On Sat, 7 May 2022 at 06:18, Chao Sun  wrote:
>>> 
>>> Seems there are some issues with the shaded client as I was not able
>>> to compile Apache Spark with the RC
>>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
>>> contain any class:
>>> 
>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
>>> META-INF/
>>> META-INF/MANIFEST.MF
>>> META-INF/NOTICE.txt
>>> META-INF/LICENSE.txt
>>> META-INF/maven/
>>> META-INF/maven/org.apache.hadoop/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>>> 
>>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
>>>> 
>>>> +1 (binding)
>>>> 
>>>>  * Signature: ok
>>>>  * Checksum : passed
>>>>  * Rat check (1.8.0_191): passed
>>>>   - mvn clean apache-rat:check
>>>>  * Built from source (1.8.0_191): failed
>>>>   - mvn clean install  -DskipTests
>>>>   - mvn -fae --no-transfer-progress -DskipTests -Dmaven.javadoc.skip=true
>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
>>>>  * Unit tests pass (1.8.0_191):
>>>>- HDFS Tests passed (Didn't run more than this).
>>>> 
>>>> Deployed a ten node ha hdfs cluster with three namenodes and five
>>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>>>> against 3.3.2) against it. Tried a small verification job. Good. Ran a
>>>> bigger job with mild chaos. All seems to be working properly (recoveries,
>>>> logs look fine). Killed a namenode. Failover worked promptly. UIs look
>>>> good. Poked at the hdfs cli. Seems good.
>>>> 
>>>> S
>>>> 
>>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran 
>>>> wrote:
>>>> 
>>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
>>>>> 
>>>>> The RC is available at:
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>>>>> 
>>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
>>>>> 
>>>>> The maven artifacts are staged at
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>>>> 
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>>> 
>>>>> Change log
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>>>>> 
>>>>> Release notes
>>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>>>>> 
>>>>> There's a very small number of changes, primarily critical code/packaging
>>>>> issues and security fixes.
>>>>> 
>>>>> 
>>>>>   - The critical fixes which shipped in the 3.2.3 release.
>>>>>   -  CVEs in our code and dependencies
>>>>>   - Shaded client packaging issues.
>>>>>   - A switch from log4j to reload4j
>>>>> 
>>>>> 
>>>>> reload4j is an active fork of the log4j 1.17 library with the classes 
>>>>> which
>>>

Re: [VOTE] Release Apache Hadoop 3.3.3

[Ayush Saxena]

There were two 3.3.3 staged. The earlier one was with skipShade, the date
was also april 22, I archived that. Chao can you use the one that Steve
mentioned in the mail?

On Sat, 7 May 2022 at 06:18, Chao Sun  wrote:

> Seems there are some issues with the shaded client as I was not able
> to compile Apache Spark with the RC
> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> contain any class:
>
> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> META-INF/
> META-INF/MANIFEST.MF
> META-INF/NOTICE.txt
> META-INF/LICENSE.txt
> META-INF/maven/
> META-INF/maven/org.apache.hadoop/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
>
> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> >
> > +1 (binding)
> >
> >   * Signature: ok
> >   * Checksum : passed
> >   * Rat check (1.8.0_191): passed
> >- mvn clean apache-rat:check
> >   * Built from source (1.8.0_191): failed
> >- mvn clean install  -DskipTests
> >- mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
> > -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > -Drequire.zstd -Drequire.test.libhadoop clean install
> >   * Unit tests pass (1.8.0_191):
> > - HDFS Tests passed (Didn't run more than this).
> >
> > Deployed a ten node ha hdfs cluster with three namenodes and five
> > journalnodes. Ran a ten node hbase (older version of 2.5 branch built
> > against 3.3.2) against it. Tried a small verification job. Good. Ran a
> > bigger job with mild chaos. All seems to be working properly (recoveries,
> > logs look fine). Killed a namenode. Failover worked promptly. UIs look
> > good. Poked at the hdfs cli. Seems good.
> >
> > S
> >
> > On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> 
> > wrote:
> >
> > > I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >
> > > The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >
> > > Release notes
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > >
> > >- The critical fixes which shipped in the 3.2.3 release.
> > >-  CVEs in our code and dependencies
> > >- Shaded client packaging issues.
> > >- A switch from log4j to reload4j
> > >
> > >
> > > reload4j is an active fork of the log4j 1.17 library with the classes
> which
> > > contain CVEs removed. Even though hadoop never used those classes, they
> > > regularly raised alerts on security scans and concen from users.
> Switching
> > > to the forked project allows us to ship a secure logging framework. It
> will
> > > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > > our log4j artifacts, as they need to cut the new dependency instead/as
> > > well.
> > >
> > > See the release notes for details.
> > >
> > > This is my first release through the new docker build process, do
> please
> > > validate artifact signing  to make sure it is good. I'll be trying
> builds
> > > of downstream projects.
> > >
> > > We know there are some outstanding issues with at least one library we
> are
> > > shipping (okhttp), but I don't want to hold this release up for it. If
> the
> > > docker based release process works smoothly enough we can do a followup
> > > security release in a few weeks.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.3

[Ayush Saxena]

+1,
* Built from source
* Successful native build on Ubuntu 18.04
* Verified Checksums
(CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
* Successful RAT check
* Ran some basic HDFS shell commands
* Ran some basic YARN shell commands
* Browsed through UI (NN ,DN, RM, NM & JHS)
* Tried some commands on Hive using Hive built on hive-master
* Verified Signature: Says Good Signature but "This key is not certified
with a trusted signature!"
* Ran some MR example jobs(TeraGen, TeraSort, TeraValidate, WordCount,
WordMean & Pi)
* Version & commit hash seems correct in UI as well as in hadoop version
output.
* Browsed through the ChangeLog & Release Notes (One place mentions hadoop
3.4.0 though, but we can survive I suppose)
* Browsed through the documentation.

Thanx Steve for driving the release, Good Luck!!!

-Ayush



On Tue, 3 May 2022 at 16:54, Steve Loughran 
wrote:

> I have put together a release candidate (rc0) for Hadoop 3.3.3
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
>
> The git tag is release-3.3.3-RC0, commit d37586cbda3
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
>
> Release notes
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
>
>- The critical fixes which shipped in the 3.2.3 release.
>-  CVEs in our code and dependencies
>- Shaded client packaging issues.
>- A switch from log4j to reload4j
>
>
> reload4j is an active fork of the log4j 1.17 library with the classes which
> contain CVEs removed. Even though hadoop never used those classes, they
> regularly raised alerts on security scans and concen from users. Switching
> to the forked project allows us to ship a secure logging framework. It will
> complicate the builds of downstream maven/ivy/gradle projects which exclude
> our log4j artifacts, as they need to cut the new dependency instead/as
> well.
>
> See the release notes for details.
>
> This is my first release through the new docker build process, do please
> validate artifact signing  to make sure it is good. I'll be trying builds
> of downstream projects.
>
> We know there are some outstanding issues with at least one library we are
> shipping (okhttp), but I don't want to hold this release up for it. If the
> docker based release process works smoothly enough we can do a followup
> security release in a few weeks.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Ayush Saxena]

>From functional point of view it does makes sense to validate all the
platforms as part of the builds, but the Pre commits builds taking time is
now no longer a small things, In past one year or may be two, we have
already increased it more than twice as compared to what it was before, If
someone has a change in HDFS, which includes both hdfs-client &
hadoop-hdfs, it takes more than 5 hours, which long back was around 2 hours.
With the current state, I don't think we should just go and add these extra
overheads. Having them as part of the nightly builds does makes sense for
now.

In future if we feel there is a strong need for this and we start to see
very frequent failures in some other platforms and we are left with no
other option but to integrate it in our pre-commit jobs, we can explore
having these build phases running in parallel, along with trying other
phases also to run in parallel like compilation/javadoc builds of JDK-8 &
JDK-11 can run in parallel and may be explore other opportunities as well
to compensate for this time.

For now lets just integrate it our nightly builds only and circle back
again here in future if the need be.

-Ayush

On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang  wrote:

> Running builds for all platforms for each and every PR seems too excessive.
>
> How about doing all platform builds in the nightly jobs?
>
> On Fri, May 6, 2022 at 8:02 AM Steve Loughran  >
> wrote:
>
> > I'm not enthusiastic here as it not only makes the builds slower, it
> > reduces the #of builds we can through a day
> >
> > one thing I am wondering is could we remove java8 support on some
> branches?
> >
> > make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and
> this
> > summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11
> > only.
> > that would cut buld and test time for those trunk PRs in half...after
> which
> > the preospect of building on more than one platform becomes more viable.
> >
> > On Thu, 5 May 2022 at 15:34, Gautham Banasandra 
> > wrote:
> >
> > > Hi Hadoop devs,
> > >
> > > Last week, there was a Hadoop build failure on Debian 10 caused by
> > > https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh,
> > > there's the capability to build and test Hadoop across the supported
> > > platforms. Currently, we're limiting this only for those PRs having
> only
> > > C/C++ changes[1], since C/C++ changes are more likely to cause
> > > cross-platform build issues and bypassing the full platform build for
> non
> > > C/C++ PRs would save a great deal of CI time. However, the build
> failure
> > > caused by PR #3988 motivates me to enable the capability to build and
> > > test Hadoop for all the supported platforms for ALL the PRs.
> > >
> > > While this may cause longer CI run duration for each PR, it would
> > > immensely minimize the risk of breaking Hadoop across platforms and
> > > saves us a lot of debugging time. Kindly post your opinion regarding
> this
> > > and I'll move to enable this capability for all PRs if the response is
> > > sufficiently positive.
> > >
> > > [1] =
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103
> > >
> > >
> > > Thanks,
> > > --Gautham
> > >
> >
>

Fwd: Applications for Travel Assistance to ApacheCon NA 2022 now open

[Ayush Saxena]

Forwarded as asked in the mail.

-Ayush

Begin forwarded message:

> From: Gavin McDonald 
> Date: 4 April 2022 at 1:56:40 PM IST
> To: travel-assista...@apache.org
> Subject: Applications for Travel Assistance to ApacheCon NA 2022 now open
> Reply-To: tac-ap...@apache.org
> 
> Hello, Apache PMCs!
> 
> Please redistribute the below to your user and dev lists, feel free to also
> use social media to spread the word. Thanks!
> 
> ---
> 
> The ASF Travel Assistance Committee (TAC) is pleased to announce that travel
> assistance applications for ApacheCon NA 2022 are now open!
> 
> We will be supporting ApacheCon North America in New Orleans, Louisiana,
> on October 3rd through 6th, 2022.
> 
> TAC exists to help those that would like to attend ApacheCon events, but
> are unable to do so for financial reasons. This year, We are supporting
> both committers and non-committers involved with projects at the
> Apache Software Foundation, or open source projects in general.
> 
> For more info on this year's applications and qualifying criteria, please
> visit the TAC website at http://www.apache.org/travel/
> Applications opened today and will close on the 1st of July 2022.
> 
> Important: Applicants have until the closing date above to submit their
> applications (which should contain as much supporting material as required
> to efficiently and accurately process their request), this will enable TAC
> to announce successful awards shortly afterwards.
> 
> As usual, TAC expects to deal with a range of applications from a diverse
> range of backgrounds. We therefore encourage (as always) anyone thinking
> about sending in an application to do so ASAP.

Re: [DISCUSS] Race condition in ProtobufRpcEngine2

[Ayush Saxena]

Hey Andras,
I had a quick look at HADOOP-18143, the methods in question in
ProtobufRpcEngine2 are identical to the ones in ProtobufRpcEngine. So, I am
not very sure how the  race condition doesn't happen in  ProtobufRpcEngine.
I have to debug and spend some more time, considering that I have
reverted HADOOP-18082 for now to unblock YARN. Though the issue would still
be there as you said, but will give us some time to analyse.

Thanks
-Ayush

On Mon, 28 Feb 2022 at 21:26, Gyori Andras 
wrote:

> Hey everyone!
>
> We have started seeing test failures in YARN PRs for a while. We have
> identified the problematic commit, which is HADOOP-18082
> , however, this change
> just revealed the race condition lying in ProtobufRpcEngine2 introduced in
> HADOOP-17046 . We have
> also fixed the underlying issue via a locking mechanism, presented in
> HADOOP-18143 , but
> since it is out of our area of expertise, we can neither verify nor
> guarantee that it will not cause some subtle issues in the RPC system.
> As we think it is a core part of Hadoop, we would use feedback from someone
> who is proficient in this part.
>
> Regards:
> Andras
>

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

[Ayush Saxena]

Moved to Dev lists.

Not sure about this though:
 when a PR is submitted to Nutch project it will run some MR job in Hadoop CI.

Whatever that PR requires should run as part of Nutch Infra. Why in Hadoop CI?
Our CI is already loaded with our own workloads.
If by any chance the above assertion gets a pass, then secondly we have very 
less number of people managing work related to CI and Infra. I don’t think most 
of the people won’t have context or say in the Nutch project, neither bandwidth 
to fix stuff if it gets broken.

Just my thoughts. Looped in the dev lists, if others have any feedback. As for 
the process, this would require a consensus from the Hadoop PMC

-Ayush

> On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney  wrote:
> 
> Hi general@,
> 
> Not sure if this is the correct mailing list. Please redirect me if there
> is a more suitable location. Thank you
> 
> I am PMC over on the Nutch project (https://nutch.apache.org). I would like
> to investigate whether we can build an integration testing capability for
> the project. This would involve running a Nutch integration test suite
> (collection of MR jobs) in a Hadoop CI environment. For example whenever a
> pull request is submitted to the Nutch project. This could easily be
> automated through Jenkins.
> 
> I’m not sure if this is something the Hadoop PMC would consider. Thank you
> for the consideration.
> 
> lewismc
> -- 
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging PRs vs. commit from CLI and keep committer field

[Ayush Saxena]

Hey Sean,
I just gave a try to Github CLI and merged a PR, using the command:

*gh pr merge 3600 --squash --body "HDFS-16290.--Message--"*


Unfortunately this also has the same result, nothing different.


-Ayush

On Tue, 26 Oct 2021 at 21:25, Sean Busbey  wrote:

> If you add a line in the commit message that the commit closes a given PR
> # then GitHub will annotate the PR as related to the specific commit and
> close it for you.
>
> i.e. you can add “closes #3454” to the commit message body and then PR
> 3454 will close and link to that commit when it hits the default branch.
>
> I believe these docs describing associating a GitHub issue via a commit
> message also apply to associating commits to pull requests, if you are
> interested in what specific phrases work:
>
>
> https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
>
>
> Has anyone tried handling a squash merge via the GitHub CLI tooling rather
> than the web or plain git tooling? Does it similarly overwrite committer
> metadata?
>
> e.g. the GitHub CLI example from the merging PRs docs?
>
>
> https://docs.github.com/en/github/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request#merging-a-pull-request
>
>
>
> On Oct 25, 2021, at 9:57 AM, Szilárd Németh  wrote:
>
> Hi Ayush,
> Thanks a lot for your response.
>
>
> Yahh, remember chasing Github support regarding this last year and they
> reverted back that they are aware of this and have an internal jira for
> this but can not comment upon how much time it would take. (As of now 1
> year & counting)
>
> So if I get this right: Their internal jira is in the same untouched state
> and they wouldn't make any progress?
>
>
> Just in case you want to use the CLI & still make the PR marked as merged,
>
> A dirty way to do this is:
> Pull the changes to your local.
> Rebase & Squash all commits, in the commit message in the end put actual
> the PR number, eg. #1234,
> force push to the PR, that is the contributors fork, this will update his
> PR and then merge the Branch to trunk in your local and push. It marks
> the PR as merged, at least last year it did when I tried. :-)
>
>
> Thanks for this dirty hack :)
> Probably I will refrain from doing this, I don't like to force push if it's
> not "mandatory".
> Anyway, if the community is fine with it, I will continue to commit from
> the CLI and close the PR, even though it's not that convenient.
>
>
> Best,
> Szilard
>
>
>
> On Wed, 20 Oct 2021 at 05:07, Ayush Saxena  wrote:
>
> Yahh, remember chasing Github support regarding this last year and they
> reverted back that they are aware of this and have an internal jira for
> this but can not comment upon how much time it would take. (As of now 1
> year & counting)
>
> As far as I remember this is with squash & commit only. If you do say a
> rebase & merge. The commit email id is preserved. But other options we have
> blocked.
>
> I think most of the projects are using squash & merge and many people
> won’t be ok to use CLI rather than UI
> So, I don’t think we have an ALT here.
>
> Just in case you want to use the CLI & still make the PR marked as merged,
> A dirty way to do this is:
> Pull the changes to your local.
> Rebase & Squash all commits, in the commit message in the end put actual
> the PR number, eg. #1234,
> force push to the PR, that is the contributors fork, this will update his
> PR and then merge the Branch to trunk in your local and push. It marks the
> PR as merged, at least last year it did when I tried. :-)
>
> -Ayush
>
>
> On 20-Oct-2021, at 3:27 AM, Szilárd Németh  wrote:
>
> Hi,
>
> I noticed something strange in our commits, in particular the committer
> field is not reflecting the user who committed the commit.
>
> *1. First, I wanted to check Gergely's commits from the last month or so.
> This was getting to be suspicious as I expected to see a bunch of commits
> from Sept / Oct of this year. *
>
> *git log CLI output:*
> ➜ git --no-pager log --format=fuller --committer=shuzirra
> commit 44bab51be44e31224dabbfa548eb27ea5fb2f916
> Author: Gergely Pollak 
> AuthorDate: Wed Aug 4 15:43:07 2021 +0200
> Commit: Gergely Pollak 
> CommitDate: Wed Aug 4 15:43:57 2021 +0200
>
>
>   YARN-10849 Clarify testcase documentation for
> TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by
> Szilard Nemeth
>
>
> commit e9339aa3761295fe65bb786e01938c7c177cd6e7
> Author: Gergely Pollak 
> AuthorDate: Tue Jun 1 15:57:22 2021

Re: [DISCUSS] Merging PRs vs. commit from CLI and keep committer field

[Ayush Saxena]

Yahh, remember chasing Github support regarding this last year and they 
reverted back that they are aware of this and have an internal jira for this 
but can not comment upon how much time it would take. (As of now 1 year & 
counting)

As far as I remember this is with squash & commit only. If you do say a rebase 
& merge. The commit email id is preserved. But other options we have blocked. 

I think most of the projects are using squash & merge and many people won’t be 
ok to use CLI rather than UI
So, I don’t think we have an ALT here.

Just in case you want to use the CLI & still make the PR marked as merged, A 
dirty way to do this is:
Pull the changes to your local.
Rebase & Squash all commits, in the commit message in the end put actual the PR 
number, eg. #1234, 
force push to the PR, that is the contributors fork, this will update his PR 
and then merge the Branch to trunk in your local and push. It marks the PR as 
merged, at least last year it did when I tried. :-)

-Ayush


> On 20-Oct-2021, at 3:27 AM, Szilárd Németh  wrote:
> 
> Hi,
> 
> I noticed something strange in our commits, in particular the committer
> field is not reflecting the user who committed the commit.
> 
> *1. First, I wanted to check Gergely's commits from the last month or so.
> This was getting to be suspicious as I expected to see a bunch of commits
> from Sept / Oct of this year. *
> 
> *git log CLI output:*
> ➜ git --no-pager log --format=fuller --committer=shuzirra
> commit 44bab51be44e31224dabbfa548eb27ea5fb2f916
> Author: Gergely Pollak 
> AuthorDate: Wed Aug 4 15:43:07 2021 +0200
> Commit: Gergely Pollak 
> CommitDate: Wed Aug 4 15:43:57 2021 +0200
> 
> 
>YARN-10849 Clarify testcase documentation for
> TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by
> Szilard Nemeth
> 
> 
> commit e9339aa3761295fe65bb786e01938c7c177cd6e7
> Author: Gergely Pollak 
> AuthorDate: Tue Jun 1 15:57:22 2021 +0200
> Commit: Gergely Pollak 
> CommitDate: Tue Jun 1 15:57:22 2021 +0200
> 
> 
>YARN-10797. Logging parameter issues in scheduler package. Contributed
> by Szilard Nemeth
> 
> 
> *2. Another example of a merged PR, here I was the author and Adam Antal
> was the committer:  *
> PR link: https://github.com/apache/hadoop/pull/3454
> 
> *git log CLI output:*
> ➜ git --no-pager log --format=fuller a9b2469a534 -1
> commit a9b2469a534c5bc554c09aaf2d460a5a00922aca
> Author: Adam Antal 
> AuthorDate: Sun Sep 19 14:42:02 2021 +0200
> Commit: GitHub 
> CommitDate: Sun Sep 19 14:42:02 2021 +0200
> 
> 
>YARN-10950. Code cleanup in QueueCapacities (#3454)
> 
> 
> *3. Let's see another two example of merged PRs by Gergely and how the git
> log CLI output look like for these commits: *
> 
> *3.1.*
> PR link: https://github.com/apache/hadoop/pull/3419
> Commit:
> https://github.com/apache/hadoop/commit/4df4389325254465b52557d6fa99bcd470d64409
> 
> *git log CLI output:*
> ➜ git --no-pager log --format=fuller
> 4df4389325254465b52557d6fa99bcd470d64409 -1
> commit 4df4389325254465b52557d6fa99bcd470d64409
> Author: Szilard Nemeth <954799+szilard-nem...@users.noreply.github.com>
> AuthorDate: Mon Sep 20 16:47:46 2021 +0200
> Commit: GitHub 
> CommitDate: Mon Sep 20 16:47:46 2021 +0200
> 
> 
>YARN-10911. AbstractCSQueue: Create a separate class for usernames and
> weights that are travelling in a Map. Contributed by Szilard Nemeth
> 
> 
> *3.2.  *
> PR link: https://github.com/apache/hadoop/pull/3342
> Commit:
> https://github.com/apache/hadoop/commit/9f6430c9ed2bca5696e77bfe9eda5d4f10b0d280
> 
> *git log CLI output:*
> ➜ git --no-pager log --format=fuller
> 9f6430c9ed2bca5696e77bfe9eda5d4f10b0d280 -1
> commit 9f6430c9ed2bca5696e77bfe9eda5d4f10b0d280
> Author: 9uapaw 
> AuthorDate: Tue Sep 21 16:08:24 2021 +0200
> Commit: GitHub 
> CommitDate: Tue Sep 21 16:08:24 2021 +0200
> 
> 
>YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
> 
> 
> As you can see, the committer field contains: *"GitHub  >".*
> Is this something specific to Hadoop or our Gitbox commit environment?
> Basically, any PR merged on the github.com UI will lose the committer
> information in the commit, which is very bad.
> 
> As I think reviewing and having discussion on Github's UI is way better
> than in jira, the only thing that makes sense for me to do perform as a
> workaround is that downloading the patch from Github before the commit,
> then commit from the CLI by adding the author info, optionally appending
> the standard "Contributed by " message to the commit message.
> For example:
> 
> git commit -m "YARN-xxx.  Contributed by  name>" --author=
> 
> This way, both the author and committer field will be correct. One downside
> is that the PR won't be merged on Github, it will be in closed state
> because the commit is committed from the CLI, so the Github PR will have a
> misleading status.
> 
> What do you think?
> What is your workflow for commits?
> 
> 
> Thanks,
> Szilard

Re: [DISCUSS] Checkin Hadoop code formatter

[Ayush Saxena]

Thanx Viraj for initiating, Makes sense to me to include a formmater inline 
with our checkstyle rules in the code, would make life simpler for all devs.

-Ayush

> On 12-Sep-2021, at 12:28 AM, Viraj Jasani  wrote:
> 
> + common-...@hadoop.apache.org
> 
> -- Forwarded message -
> From: Viraj Jasani 
> Date: Tue, Sep 7, 2021 at 6:18 PM
> Subject: Checkin Hadoop code formatter
> To: common-...@hadoop.apache.org 
> 
> 
> It seems some recent new devs are not familiar with the common code
> formatter that we use for our codebase.
> While we already have Wiki page [1] for new contributors and it mentions:
> "Code must be formatted according to Sun's conventions
> "
> but this Oracle's code conventions page is not being actively maintained
> (no update has been received after 1999) and hence, I believe we should
> check-in and maintain code formatter xmls for supported IDEs in our
> codebase only (under dev-support) for all devs to be able to import it in
> the respective IDE.
> Keeping this in mind, I have created this PR 3387
> . If you could please take a
> look and if the PR receives sufficient +1s, we might want to update our
> Wiki page to directly refer to our own codebase for code formatters that we
> maintain. Thoughts?
> 
> 
> 1.
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-MakingChanges

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.2 release?

[Ayush Saxena]

+1,
Thanx Chao for volunteering!!!

> On 07-Sep-2021, at 10:36 PM, Chao Sun  wrote:
> 
> Hi all,
> 
> It has been almost 3 months since the 3.3.1 release and branch-3.3 has
> accumulated quite a few commits (118 atm). In particular, Spark community
> recently found an issue which prevents one from using the shaded Hadoop
> client together with certain compression codecs such as lz4 and snappy
> codec. The details are recorded in HADOOP-17891 and SPARK-36669.
> 
> Therefore, I'm wondering if anyone is also interested in a 3.3.2 release.
> If there is no objection, I'd like to volunteer myself for the work as well.
> 
> Best Regards,
> Chao

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Yetus unable to build YARN / HADOOP for any patch

[Ayush Saxena]

Got it sorted,made changes in HDFS-PreCommit and triggered the build:

https://ci-hadoop.apache.org/view/Hadoop/job/PreCommit-HDFS-Build/669/console

Got the result:

https://issues.apache.org/jira/browse/HDFS-16101?focusedCommentId=17374991=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17374991


This worked, but I have also changed the user to Hudson from the original 
hadoopci as of now.
May be I am missing something or the creds got screwed up. Need to check that.

For the time being we can use Hudson? If so we can do the same changes in all 
other pre commit jobs as well and see if things work.  I can check and get the 
creds stuff sorted out in a couple of days…

-Ayush

> On 05-Jul-2021, at 6:31 PM, Ayush Saxena  wrote:
> Yeps, something is broken since last week. I tried a bunch of things on 
> HDFS-PreCommit this weekend, kept on solving issues one after the other.
> 
> https://issues.apache.org/jira/browse/HADOOP-17787?focusedCommentId=17374212=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17374212
> 
> First it was the hadoop.sh path was coming wrong, I fixed it, post that it 
> wasn’t able to download the patch, added jira as plugin, then the curl 
> started failing with 22 error code, the hadoop ci creds were being rejected. 
> I changed it to Hudson for trying, it downloaded the patch successfully. Post 
> that it started giving error :
> 
> Unprocessed flag(s): --brief-report-file --html-report-file 
> --mvn-custom-repos  --shelldocs --mvn-javadoc-goals --mvn-custom-repos-dir
> 
> https://ci-hadoop.apache.org/view/Hadoop/job/PreCommit-HDFS-Build/664/console
> 
> Here I left it. Not sure what broke it and whether INFRA folks would help us 
> on this. The errors are coming from our scripts. 
> 
> But we need to fix it somehow. Do let me know if anyone has any pointers to 
> any recent change which might affect us.
> 
> -Ayush
> 
>> On 05-Jul-2021, at 6:06 PM, Szilárd Németh  wrote:
>> Hi All,
>> 
>> We are experiencing a build issue for HADOOP / YARN, this phenomenon
>> started around last week.
>> Just reported an infra ticket to tackle this:
>> https://issues.apache.org/jira/browse/INFRA-22079
>> Do you think of anything else we can do at this point?
>> 
>> 
>> Best regards,
>> Szilard

Re: Yetus unable to build YARN / HADOOP for any patch

[Ayush Saxena]

Yeps, something is broken since last week. I tried a bunch of things on 
HDFS-PreCommit this weekend, kept on solving issues one after the other.

https://issues.apache.org/jira/browse/HADOOP-17787?focusedCommentId=17374212=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17374212

First it was the hadoop.sh path was coming wrong, I fixed it, post that it 
wasn’t able to download the patch, added jira as plugin, then the curl started 
failing with 22 error code, the hadoop ci creds were being rejected. I changed 
it to Hudson for trying, it downloaded the patch successfully. Post that it 
started giving error :

Unprocessed flag(s): --brief-report-file --html-report-file --mvn-custom-repos  
--shelldocs --mvn-javadoc-goals --mvn-custom-repos-dir

https://ci-hadoop.apache.org/view/Hadoop/job/PreCommit-HDFS-Build/664/console

Here I left it. Not sure what broke it and whether INFRA folks would help us on 
this. The errors are coming from our scripts. 

But we need to fix it somehow. Do let me know if anyone has any pointers to any 
recent change which might affect us.

-Ayush

> On 05-Jul-2021, at 6:06 PM, Szilárd Németh  wrote:
> 
> Hi All,
> 
> We are experiencing a build issue for HADOOP / YARN, this phenomenon
> started around last week.
> Just reported an infra ticket to tackle this:
> https://issues.apache.org/jira/browse/INFRA-22079
> Do you think of anything else we can do at this point?
> 
> 
> Best regards,
> Szilard

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Ayush Saxena]

+1,
Built from Source.
Successful Native Build on Ubuntu 20.04
Verified Checksums
Ran basic hdfs shell commands.
Ran simple MR jobs.
Browsed NN,DN,RM and NM UI.

Thanx Wei-Chiu for driving the release. 

-Ayush


> On 12-Jun-2021, at 1:45 AM, epa...@apache.org wrote:
> 
> +1 (binding)
> Eric
> 
> 
> On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang 
>  wrote:  
> 
> Hi community,
> 
> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
> issues have been resolved [1] again.
> 
> There are 2 additional issues resolved for RC3:
> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> HADOOP-16878
> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
> and destination are the same
> 
> There are 4 issues resolved for RC2:
> * HADOOP-17666. Update LICENSE for 3.3.1
> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> 
> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> fixes compared to hadoop-thirdparty 1.1.0:
> * HADOOP-17707. Remove jaeger document from site index.
> * HADOOP-17730. Add back error_prone
> 
> *RC tag is release-3.3.1-RC3
> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> 
> *The RC3 artifacts are at*:
> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> 
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> 
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> 
> Things I've verified:
> * all blocker issues targeting 3.3.1 have been resolved.
> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> * LICENSE and NOTICE files checked
> * RELEASENOTES and CHANGELOG
> * rat check passed.
> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
> dependency divergence. Issues are being identified but so far nothing
> blocker for Hadoop itself.
> 
> Please try the release and vote. The vote will run for 5 days.
> 
> My +1 to start,
> 
> [1] https://issues.apache.org/jira/issues/?filter=12350491
> [2]
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Hadoop 3.1.x EOL

[Ayush Saxena]

+1

-Ayush

> On 03-Jun-2021, at 11:26 PM, hemanth boyina  
> wrote:
> 
> +1
> 
> Thanks
> HemanthBoyina
> 
>> On Thu, 3 Jun 2021, 20:33 Wanqiang Ji,  wrote:
>> 
>> +1 (non-binding)
>> 
>> Wanqiang Ji
>> 
>>> On Thu, Jun 3, 2021 at 10:47 PM Sangjin Lee  wrote:
>>> 
>>> +1
>>> 
>>> On Thu, Jun 3, 2021 at 7:35 AM Sean Busbey 
>>> wrote:
>>> 
 +1
 
> On Jun 3, 2021, at 1:14 AM, Akira Ajisaka 
>> wrote:
> 
> Dear Hadoop developers,
> 
> Given the feedback from the discussion thread [1], I'd like to start
> an official vote
> thread for the community to vote and start the 3.1 EOL process.
> 
> What this entails:
> 
> (1) an official announcement that no further regular Hadoop 3.1.x
 releases
> will be made after 3.1.4.
> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
> 
> This vote will run for 7 days and conclude by June 10th, 16:00 JST
>> [2].
> 
> Committers are eligible to cast binding votes. Non-committers are
 welcomed
> to cast non-binding votes.
> 
> Here is my vote, +1
> 
> [1] https://s.apache.org/w9ilb
> [2]
 
>>> 
>> https://www.timeanddate.com/worldclock/fixedtime.html?msg=4=20210610T16=248
> 
> Regards,
> Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
 
 
 
 -
 To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: common-dev-h...@hadoop.apache.org
 
 
>>> 
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] which release lines should we still consider actively maintained?

[Ayush Saxena]

+1, to mark 3.1.x EOL.
Apache Hive does depends on 3.1.0 as of now, but due to guave upgrade on 
branch-3.1, the attempt to migrate to latest 3.1.x didn’t work for me atleast 
couple of months back. So, mostly 3.3.1 would be the only option replacing 
3.1.0 there or at worst 3.3.2 in a couple of months.


-Ayush

> On 24-May-2021, at 8:43 PM, Arpit Agarwal  
> wrote:
> 
> +1 to EOL 3.1.x at least.
> 
> 
>> On May 23, 2021, at 9:51 PM, Wei-Chiu Chuang  
>> wrote:
>> 
>> Sean,
>> 
>> For reasons I don't understand, I never received emails from your new
>> address in the mailing list. Only Akira's response.
>> 
>> I was just able to start a thread like this.
>> 
>> I am +1 to EOL 3.1.5.
>> Reason? Spark is already on Hadoop 3.2. Hive and Tez are actively working
>> to support Hadoop 3.3. HBase supports Hadoop 3.3 already. They are the most
>> common Hadoop applications so I think a 3.1 isn't that necessarily
>> important.
>> 
>> With Hadoop 3.3.1, we have a number of improvements to support a better
>> HDFS upgrade experience, so upgrading from Hadoop 3.1 should be relatively
>> easy. Application upgrade takes some effort though (commons-lang ->
>> commons-lang3 migration for example)
>> I've been maintaining the HDFS code in branch-3.1, so from a
>> HDFS perspective the branch is always in a ready to release state.
>> 
>> The Hadoop 3.1 line is more than 3 years old. Maintaining this branch is
>> getting trickier. I am +100 to reduce the number of actively maintained
>> release line. IMO, 2 Hadoop 3 lines + 1 Hadoop 2 line is a good idea.
>> 
>> 
>> 
>> For Hadoop 3.3 line: If no one beats me, I plan to make a 3.3.2 in 2-3
>> months. And another one in another 2-3 months.
>> The Hadoop 3.3.1 has nearly 700 commits not in 3.3.0. It is very difficult
>> to make/validate a maint release with such a big divergence in the code.
>> 
>> 
>>> On Mon, May 24, 2021 at 12:06 PM Akira Ajisaka >> > wrote:
>>> 
>>> Hi Sean,
>>> 
>>> Thank you for starting the discussion.
>>> 
>>> I think branch-2.10, branch-3.1, branch-3.2, branch-3.3, and trunk
>>> (3.4.x) are actively maintained.
>>> 
>>> The next releases will be:
>>> - 3.4.0
>>> - 3.3.1 (Thanks, Wei-Chiu!)
>>> - 3.2.3
>>> - 3.1.5
>>> - 2.10.2
>>> 
 Are there folks willing to go through being release managers to get more
>>> of these release lines on a steady cadence?
>>> 
>>> Now I'm interested in becoming a release manager of 3.1.5.
>>> 
 If I were to take up maintenance release for one of them which should it
>>> be?
>>> 
>>> 3.2.3 or 2.10.2 seems to be a good choice.
>>> 
 Should we declare to our downstream users that some of these lines
>>> aren’t going to get more releases?
>>> 
>>> Now I think we don't need to declare that. I believe 3.3.1, 3.2.3,
>>> 3.1.5, and 2.10.2 will be released in the near future.
>>> There are some earlier discussions of 3.1.x EoL, so 3.1.5 may be a
>>> final release of the 3.1.x release line.
>>> 
 Is there downstream facing documentation somewhere that I missed for
>>> setting expectations about our release cadence and actively maintained
>>> branches?
>>> 
>>> As you commented, the confluence wiki pages for Hadoop releases were
>>> out of date. Updated [1].
>>> 
 Do we have a backlog of work written up that could make the release
>>> process easier for our release managers?
>>> 
>>> The release process is documented and maintained:
>>> https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
>>> Also, there are some backlogs [1], [2].
>>> 
>>> [1]:
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
>>> [2]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
>>> 
>>> Thanks,
>>> Akira
>>> 
>>> On Fri, May 21, 2021 at 7:12 AM Sean Busbey 
>>> wrote:
 
 
 Hi folks!
 
 Which release lines do we as a community still consider actively
>>> maintained?
 
 I found an earlier discussion[1] where we had consensus to consider
>>> branches that don’t get maintenance releases on a regular basis end-of-life
>>> for practical purposes. The result of that discussion was written up in our
>>> wiki docs in the “EOL Release Branches” page, summarized here
 
> If no volunteer to do a maintenance release in a short to mid-term
>>> (like 3 months to 1 or 1.5 year).
 
 Looking at release lines that are still on our download page[3]:
 
 * Hadoop 2.10.z - last release 8 months ago
 * Hadoop 3.1.z - last release 9.5 months ago
 * Hadoop 3.2.z - last release 4.5 months ago
 * Hadoop 3.3.z - last release 10 months ago
 
 And then trunk holds 3.4 which hasn’t had a release since the branch-3.3
>>> fork ~14 months ago.
 
 I can see that Wei-Chiu has been actively working on getting the 3.3.1
>>> release out[4] (thanks Wei-Chiu!) but I do not see anything similar for the
>>> other release lines.
 
 We also have pages on the wiki for our project roadmap of release[5],
>>> but it 

Re: [DISCUSS] check style changes

[Ayush Saxena]

If you tend to change the line limit, It should have a proper DISCUSS and VOTE 
thread. If I remember it correct for ozone it was discussed to increase the 
limit to 100 from 80 and that got vetoed. 
So, we too should get an agreement first. Well I have gone through the ozone 
discussion and I also don’t support changing the line length limit in general.

Regarding the nightly runs, I think fixing the flaky tests and getting away 
with the frequent OOM errors in the build is something should get priority, 
that would be helpfull to the project.

Fixing checkstyles, firstly would be just a look and feel change. Many 
checkstyle warnings would be the one accepted by the committer while commiting 
and the most important stuff, it would make backports tough, checking the git 
history a little tougher, and apart from that these changes would be huge. So 
reviewers need to be extra cautious, that accidentally some bug doesn’t get 
induced. 

So, Personal opinions: Line length stuff needs proper discussion and vote, 
justifying a strong reason.

Chekstyle modifications also, if it is like tooo much of change, and no big 
advantages, I think atleast I can survive with it, giving the fact it might 
take my ease to backport issues internally as well as to other branches. But 
still since it won’t break anything, so I don’t think so, I have any right to 
say No to this. But in case you plan to pick this up with priority. Get the 
plan discussed before merging.

Would be happy to help in case you plan to pick up the other stuffs related to 
the builds. :-)

-Ayush

> On 13-May-2021, at 8:40 PM, Sean Busbey  wrote:
> 
> Hi folks!
> 
> I’d like to start cleaning up our nightly tests. As a bit of low hanging 
> fruit I’d like to alter some of our check style rules to match what I think 
> we’ve been doing in the community. How would folks prefer I make sure we have 
> consensus on such changes?
> 
> As an example, our last nightly run had ~81k check style violations (it’s a 
> big number but it’s not that bad given the size of the repo) and roughly 16% 
> of those were for line lengths in excess of 80 characters but <= 100 
> characters.
> 
> If I wanted to change our line length check to be 100 characters rather than 
> the default of 80, would folks rather I have a DISCUSS thread first? Or would 
> they rather a Jira + PR with the discussion of the merits happening there?
> 
> —
> busbey
> 
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

[Ayush Saxena]

Yep, you have to do it manually

-Ayush

> On 26-Apr-2021, at 3:23 PM, Wei-Chiu Chuang  wrote:
> 
> 
> Does anyone know how we publish hadoop-thirdparty SNAPSHOT artifacts?
> 
> The main Hadoop arifacts are published by this job 
> https://ci-hadoop.apache.org/view/Hadoop/job/Hadoop-trunk-Commit/ after every 
> commit.
> However, we don't seem to publish hadoop-thirdparty regularly. (Apache nexus: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/thirdparty/)
> 
> Are they published manually?
> 
> 
>> On Fri, Apr 23, 2021 at 6:06 PM Ayush Saxena  wrote:
>> Regarding Guava: before release once you merge the change to thrirdparty 
>> repo, can update the hadoop thirdparty snapshot, the hadoop code would pick 
>> that up, and watch out everything is safe and clean before release. Unless 
>> you have a better way to verify or already verified!!! 
>> 
>> -Ayush
>> 
>> > On 23-Apr-2021, at 3:16 PM, Wei-Chiu Chuang  
>> > wrote:
>> > 
>> > Another suggestion: looks like the shaded jaeger is not being used by
>> > Hadoop code. Maybe we can remove that from the release for now? I don't
>> > want to release something that's not being used.
>> > We can release the shaded jaeger when it's ready for use. We will have to
>> > update the jaeger version anyway. The version used is too old.
>> > 
>> >> On Fri, Apr 23, 2021 at 10:55 AM Wei-Chiu Chuang  
>> >> wrote:
>> >> 
>> >> Hi community,
>> >> 
>> >> In preparation of the Hadoop 3.3.1 release, I am starting a thread to
>> >> discuss its prerequisite: the release of hadoop-thirdparty 1.1.0.
>> >> 
>> >> My plan:
>> >> update guava to 30.1.1 (latest). I have the PR ready to merge.
>> >> 
>> >> Do we want to update protobuf and jaeger? Anything else?
>> >> 
>> >> I suppose we won't update protobuf too frequently.
>> >> Jaeger is under active development. We're currently on 0.34.2, the latest
>> >> is 1.22.0.
>> >> 
>> >> If there is no change to this plan, I can start the release work as soon
>> >> as possible.
>> >> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 

Re: [DISCUSS] hadoop-thirdparty 1.1.0 release

[Ayush Saxena]

Regarding Guava: before release once you merge the change to thrirdparty repo, 
can update the hadoop thirdparty snapshot, the hadoop code would pick that up, 
and watch out everything is safe and clean before release. Unless you have a 
better way to verify or already verified!!! 

-Ayush

> On 23-Apr-2021, at 3:16 PM, Wei-Chiu Chuang  
> wrote:
> 
> Another suggestion: looks like the shaded jaeger is not being used by
> Hadoop code. Maybe we can remove that from the release for now? I don't
> want to release something that's not being used.
> We can release the shaded jaeger when it's ready for use. We will have to
> update the jaeger version anyway. The version used is too old.
> 
>> On Fri, Apr 23, 2021 at 10:55 AM Wei-Chiu Chuang  wrote:
>> 
>> Hi community,
>> 
>> In preparation of the Hadoop 3.3.1 release, I am starting a thread to
>> discuss its prerequisite: the release of hadoop-thirdparty 1.1.0.
>> 
>> My plan:
>> update guava to 30.1.1 (latest). I have the PR ready to merge.
>> 
>> Do we want to update protobuf and jaeger? Anything else?
>> 
>> I suppose we won't update protobuf too frequently.
>> Jaeger is under active development. We're currently on 0.34.2, the latest
>> is 1.22.0.
>> 
>> If there is no change to this plan, I can start the release work as soon
>> as possible.
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.1 release

[Ayush Saxena]

+1
Just to mention we would need to release hadoop-thirdparty too before. 
Presently we are using the snapshot version of it.

-Ayush

> On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang  wrote:
> 
> Hi all,
> 
> Hadoop 3.3.0 was released half a year ago, and as of now we've accumulated
> more than 400 changes in the branch-3.3. A number of downstreamers are
> eagerly waiting for 3.3.1 which addresses the guava version conflict issue.
> 
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> 
> We should start the release work for 3.3.1 before the diff becomes even
> larger.
> 
> I believe there are  currently only two real blockers for a 3.3.1 (using
> this filter
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> )
> 
> 
>   1. HDFS-15566 
>   2.
>  1. HADOOP-17112 
> 2.
> 
> 
> 
> Is there anyone who would volunteer to be the 3.3.1 RM?
> 
> Also, the HowToRelease wiki does not describe the ARM build process. That's
> going to be important for future releases.

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Intervention. Stabilizing Yetus (Attn. Azure)

[Ayush Saxena]

The failing Azure tests are being tracked at HADOOP-17325

https://issues.apache.org/jira/browse/HADOOP-17325

On Mon, 9 Nov 2020 at 23:02, Ahmed Hussein  wrote:

> I created new Jiras for HDFS failures. Please consider doing the same for
> Yarn and Azure.
> For convenience, the list of failures in the qbt report is as follows:
>
> Test Result
> <
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/
> >
> (50
> failures / -7)
>
>-
>
>  
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testGetCachedDatanodeReport
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testGetCachedDatanodeReport/
> >
>-
>
>  
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testNamenodeMetrics
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testNamenodeMetrics/
> >
>-
>
>  
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testErasureCoding
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testErasureCoding/
> >
>-
>
>  
> org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> >
>-
> org.apache.hadoop.yarn.sls.TestReservationSystemInvariants.testSimulatorRunning[Testing
>with: SYNTH,
>
>  org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
>(nodeFile null)]
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls/TestReservationSystemInvariants/testSimulatorRunning_Testing_with__SYNTH__org_apache_hadoop_yarn_server_resourcemanager_scheduler_fair_FairScheduler___nodeFile_null__/
> >
>-
> org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator.testAMSimulator[1]
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls.appmaster/TestAMSimulator/testAMSimulator_1_/
> >
>-
>
>  
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.server.resourcemanager.security/TestDelegationTokenRenewer/testTokenThreadTimeout/
> >
>-
>
>  
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithOpportunisticContainers/
> >
>-
>
>  
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithEnforceExecutionType/
> >
>- org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/
> >
>-
>
>  org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/
> >
>- org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testPermissionMetadata/
> >
>- org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testOldPermissionMetadata/
> >
>-
>
>  
> org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible
><
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestNativeAzureFileSystemConcurrency/testNoTempBlobsVisible/
> >
>-
>
>  org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency.testLinkBlobs
><
> 

Re: Precommit job fails to write comments in the GitHub pull requests

[Ayush Saxena]

Hi Akira,
Thanx for working on this.
Is there anything blocking here? I think it isn’t working still.
Can we point to a commit before Yetus-994 instead of master and get away with 
this? or if there is no solution handy we may go ahead with approach #4 and 
then figure out how to proceed.

-Ayush

> On 29-Sep-2020, at 5:31 PM, Masatake Iwasaki  
> wrote:
> 
> Hi Akira,
> 
> Thanks for working on this.
> It looks like you are trying token on PR #2348.
> https://github.com/apache/hadoop/pull/2348
> 
> It it does not work, I think option #4 is reasonable.
> I prefer using fixed/static version of toolchain in order to
> avoid surprising the developers.
> We can upgrade Yetus when JDK11 Javadoc issue is fixed.
> 
> Regards,
> Masatake Iwasaki
> 
>> On 2020/09/29 2:55, Akira Ajisaka wrote:
>> Apache Hadoop developers,
>> After YETUS-994, the jenkins job updates the commit status instead of
>> writing a comment to the pull request. It requires an OAuth token with
>> write access to "repo:status" but now there is no such token for
>> Apache Hadoop. I asked the infra team to create a token:
>> https://issues.apache.org/jira/browse/INFRA-20906
>> For now, I think there 4 options to get the information:
>> 1. Attach a patch to the JIRA instead of GitHub
>> 2. Update the Jenkinsfile for a while (Sample PR:
>> https://github.com/apache/hadoop/pull/2346)
>> 3. Search the job and get the comment from
>> https://ci-hadoop.apache.org/job/hadoop-multibranch/
>> 4. Create a JIRA to update the Jenkinsfile to use 0.12.0 instead of
>> main. (JDK11 javadoc support is broken in Yetus 0.12.0)
>> Sorry for the inconvenience.
>> Regards,
>> Akira
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Apache Hadoop Ozone 1.0.0 RC1

[Ayush Saxena]

+1
* Built from source
* Verified checksums & signature
* Ran some basic shell commands.

Thanx Sammi for driving the release. Good Luck!!!

-Ayush

On Tue, 1 Sep 2020 at 13:20, Mukul Kumar Singh 
wrote:

> Thanks for preparing the RC Sammi.
>
> +1 (binding)
>
> 1. Verified Signatures
>
> 2. Compiled the source
>
> 3. Local Docker based deployed clsuter and some basic commands.
>
> Thanks,
>
> Mukul
>
> On 01/09/20 12:07 pm, Rakesh Radhakrishnan wrote:
> > Thanks Sammi for getting this out!
> >
> > +1 (binding)
> >
> >   * Verified signatures.
> >   * Built from source.
> >   * Deployed small non-HA un-secure cluster.
> >   * Verified basic Ozone file system.
> >   * Tried out a few basic Ozone shell commands - create, list, delete
> >   * Ran a few Freon benchmark tests.
> >
> > Thanks,
> > Rakesh
> >
> > On Tue, Sep 1, 2020 at 11:53 AM Jitendra Pandey
> >  wrote:
> >
> >> +1 (binding)
> >>
> >> 1. Verified signatures
> >> 2. Built from source
> >> 3. deployed with docker
> >> 4. tested with basic s3 apis.
> >>
> >> On Tue, Aug 25, 2020 at 7:01 AM Sammi Chen 
> wrote:
> >>
> >>> RC1 artifacts are at:
> >>> https://home.apache.org/~sammichen/ozone-1.0.0-rc1/
> >>> 
> >>>
> >>> Maven artifacts are staged at:
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1278
> >>> <
> https://repository.apache.org/content/repositories/orgapachehadoop-1277
> >>>
> >>>
> >>> The public key used for signing the artifacts can be found at:
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>
> >>> The RC1 tag in github is at:
> >>> https://github.com/apache/hadoop-ozone/releases/tag/ozone-1.0.0-RC1
> >>> 
> >>>
> >>> Change log of RC1, add
> >>> 1. HDDS-4063. Fix InstallSnapshot in OM HA
> >>> 2. HDDS-4139. Update version number in upgrade tests.
> >>> 3. HDDS-4144, Update version info in hadoop client dependency readme
> >>>
> >>> *The vote will run for 7 days, ending on Aug 31th 2020 at 11:59 pm
> PST.*
> >>>
> >>> Thanks,
> >>> Sammi Chen
> >>>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] End of Life Hadoop 2.9

[Ayush Saxena]

+1

-Ayush

> On 01-Sep-2020, at 7:23 AM, Akira Ajisaka  wrote:
> 
> +1
> 
> -Akira
> 
>> On Tue, Sep 1, 2020 at 4:09 AM Wei-Chiu Chuang  wrote:
>> 
>> Dear fellow Hadoop developers,
>> 
>> Given the overwhelming feedback from the discussion thread
>> https://s.apache.org/hadoop2.9eold, I'd like to start an official vote
>> thread for the community to vote and start the 2.9 EOL process.
>> 
>> What this entails:
>> 
>> (1) an official announcement that no further regular Hadoop 2.9.x releases
>> will be made after 2.9.2 (which was GA on 11/19/2019)
>> (2) resolve JIRAs that specifically target 2.9.3 as won't fix.
>> 
>> 
>> This vote will run for 7 days and will conclude by September 7th, 12:00pm
>> pacific time.
>> Committers are eligible to cast binding votes. Non-committers are welcomed
>> to cast non-binding votes.
>> 
>> Here is my vote, +1
>> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] fate of branch-2.9

[Ayush Saxena]

+1

-Ayush

> On 27-Aug-2020, at 6:24 PM, Steve Loughran  
> wrote:
> 
> 
> 
> +1
> 
> are there any Hadoop branch-2 releases planned, ever? If so I'll need to 
> backport my s3a directory compatibility patch to whatever is still live.
> 
> 
>> On Thu, 27 Aug 2020 at 06:55, Wei-Chiu Chuang  wrote:
>> Bump up this thread after 6 months.
>> 
>> Is anyone still interested in the 2.9 release line? Or are we good to start
>> the EOL process? The 2.9.2 was released in Nov 2018.
>> 
>> I'd really like to see the community to converge to fewer release lines and
>> make more frequent releases in each line.
>> 
>> Thanks,
>> Weichiu
>> 
>> 
>> On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang  wrote:
>> 
>> > I think that's a great suggestion.
>> > Currently, we make 1 minor release per year, and within each minor release
>> > we bring up 1 thousand to 2 thousand commits in it compared with the
>> > previous one.
>> > I can totally understand it is a big bite for users to swallow. Having a
>> > more frequent release cycle, plus LTS and non-LTS releases should help with
>> > this. (Of course we will need to make the release preparation much easier,
>> > which is currently a pain)
>> >
>> > I am happy to discuss the release model further in the dev ML. LTS v.s.
>> > non-LTS is one suggestion.
>> >
>> > Another similar issue: In the past Hadoop strived to
>> > maintain compatibility. However, this is no longer sustainable as more CVEs
>> > coming from our dependencies: netty, jetty, jackson ... etc.
>> > In many cases, updating the dependencies brings breaking changes. More
>> > recently, especially in Hadoop 3.x, I started to make the effort to update
>> > dependencies much more frequently. How do users feel about this change?
>> >
>> > On Thu, Mar 5, 2020 at 7:58 AM Igor Dvorzhak 
>> > wrote:
>> >
>> >> Maybe Hadoop will benefit from adopting a similar release and support
>> >> strategy as Java? I.e. designate some releases as LTS and support them for
>> >> 2 (?) years (it seems that 2.7.x branch was de-facto LTS), other non-LTS
>> >> releases will be supported for 6 months (or until next release). This
>> >> should allow to reduce maintenance cost of non-LTS release and provide
>> >> conservative users desired stability by allowing them to wait for new LTS
>> >> release and upgrading to it.
>> >>
>> >> On Thu, Mar 5, 2020 at 1:26 AM Rupert Mazzucco 
>> >> wrote:
>> >>
>> >>> After recently jumping from 2.7.7 to 2.10 without issue myself, I vote
>> >>> for keeping only the 2.10 line.
>> >>> It would seem all other 2.x branches can upgrade to a 2.10.x easily if
>> >>> they feel like upgrading at all,
>> >>> unlike a jump to 3.x, which may require more planning.
>> >>>
>> >>> I also vote for having only one main 3.x branch. Why are there 3.1.x and
>> >>> 3.2.x seemingly competing,
>> >>> and now 3.3.x? For a community that does not have the resources to
>> >>> manage multiple release lines,
>> >>> you guys sure like to multiply release lines a lot.
>> >>>
>> >>> Cheers
>> >>> Rupert
>> >>>
>> >>> Am Mi., 4. März 2020 um 19:40 Uhr schrieb Wei-Chiu Chuang
>> >>> :
>> >>>
>>  Forwarding the discussion thread from the dev mailing lists to the user
>>  mailing lists.
>> 
>>  I'd like to get an idea of how many users are still on Hadoop 2.9.
>>  Please share your thoughts.
>> 
>>  On Mon, Mar 2, 2020 at 6:30 PM Sree Vaddi
>>   wrote:
>> 
>> > +1
>> >
>> > Sent from Yahoo Mail on Android
>> >
>> >   On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang
>> > wrote:   Hi,
>> >
>> > Following the discussion to end branch-2.8, I want to start a
>> > discussion
>> > around what's next with branch-2.9. I am hesitant to use the word "end
>> > of
>> > life" but consider these facts:
>> >
>> > * 2.9.0 was released Dec 17, 2017.
>> > * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is more
>> > than
>> > 15 months ago.
>> > * no one seems to be interested in being the release manager for 2.9.3.
>> > * Most if not all of the active Hadoop contributors are using Hadoop
>> > 2.10
>> > or Hadoop 3.x.
>> > * We as a community do not have the cycle to manage multiple release
>> > line,
>> > especially since Hadoop 3.3.0 is coming out soon.
>> >
>> > It is perhaps the time to gradually reduce our footprint in Hadoop
>> > 2.x, and
>> > encourage people to upgrade to Hadoop 3.x
>> >
>> > Thoughts?
>> >
>> >

Re: [RESULT][VOTE] Rlease Apache Hadoop-3.3.0

[Ayush Saxena]

Hi Brahma,
Seems the link to changelog for Release-3.3.0 isn't correct at :
https://hadoop.apache.org/

It points to :
http://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGES.3.3.0.html

CHANGES.3.3.0.html isn't there, instead it should point to :

http://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGELOG.3.3.0.html

Please give a check once!!!

-Ayush




On Wed, 15 Jul 2020 at 19:18, Brahma Reddy Battula 
wrote:

> Hi Stephen,
>
> thanks for bringing this to my attention.
>
> Looks it's late..I pushed the release tag ( which can't be reverted) and
> updated the release date in the jira.
>
> Can we plan this next release near future..?
>
>
> On Wed, Jul 15, 2020 at 5:25 PM Stephen O'Donnell
>  wrote:
>
> > Hi All,
> >
> > Sorry for being a bit late to this, but I wonder if we have a potential
> > blocker to this release.
> >
> > In Cloudera we have recently encountered a serious dataloss issue in HDFS
> > surrounding snapshots. To hit the dataloss issue, you must have
> HDFS-13101
> > and HDFS-15012 on the build (which branch-3.3.0 does). To prevent it, you
> > must also have HDFS-15313 and unfortunately, this was only committed to
> > trunk, so we need to cherry-pick it down the active branches.
> >
> > With data loss being a serious issue, should we pull this Jira into
> > branch-3.3.0 and cut a new release candidate?
> >
> > Thanks,
> >
> > Stephen.
> >
> > On Tue, Jul 14, 2020 at 1:22 PM Brahma Reddy Battula 
> > wrote:
> >
> > > Hi All,
> > >
> > > With 8 binding and 11 non-binding +1s and no -1s the vote for Apache
> > > hadoop-3.3.0 Release
> > > passes.
> > >
> > > Thank you everybody for contributing to the release, testing, and
> voting.
> > >
> > > Special thanks whoever verified the ARM Binary as this is the first
> > release
> > > to support the ARM in hadoop.
> > >
> > >
> > > Binding +1s
> > >
> > > =
> > > Akira Ajisaka
> > > Vinayakumar B
> > > Inigo Goiri
> > > Surendra Singh Lilhore
> > > Masatake Iwasaki
> > > Rakesh Radhakrishnan
> > > Eric Badger
> > > Brahma Reddy Battula
> > >
> > > Non-binding +1s
> > >
> > > =
> > > Zhenyu Zheng
> > > Sheng Liu
> > > Yikun Jiang
> > > Tianhua huang
> > > Ayush Saxena
> > > Hemanth Boyina
> > > Bilwa S T
> > > Takanobu Asanuma
> > > Xiaoqiao He
> > > CR Hota
> > > Gergely Pollak
> > >
> > > I'm going to work on staging the release.
> > >
> > >
> > > The voting thread is:
> > >
> > >  https://s.apache.org/hadoop-3.3.0-Release-vote-thread
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
>
>
> --
>
>
>
> --Brahma Reddy Battula
>