Re: [VOTE] Hadoop 3.2.x EOL

[Akira Ajisaka]

+1



On Wed, Dec 6, 2023 at 1:10 PM Xiaoqiao He  wrote:

> Dear Hadoop devs,
>
> Given the feedback from the discussion thread [1], I'd like to start
> an official thread for the community to vote on release line 3.2 EOL.
>
> It will include,
> a. An official announcement informs no further regular Hadoop 3.2.x
> releases.
> b. Issues which target 3.2.5 will not be fixed.
>
> This vote will run for 7 days and conclude by Dec 13, 2023.
>
> I’ll start with my +1.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://lists.apache.org/thread/bbf546c6jz0og3xcl9l3qfjo93b65szr
>

[jira] [Resolved] (MAPREDUCE-7390) Remove WhiteBox in mapreduce module.

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7390.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

Committed to trunk.

> Remove WhiteBox in mapreduce module.
> 
>
> Key: MAPREDUCE-7390
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7390
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> WhiteBox is deprecated, try to remove this method in hadoop-mapreduce.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7426) Fix typo in class StartEndTImesBase

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7426.
--
Fix Version/s: 3.4.0
 Assignee: Samrat Deb
   Resolution: Fixed

Merged the PR into trunk.

> Fix typo in class StartEndTImesBase
> ---
>
> Key: MAPREDUCE-7426
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7426
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Samrat Deb
>Assignee: Samrat Deb
>Priority: Trivial
>  Labels: newbie, pull-request-available
> Fix For: 3.4.0
>
>
> While going through the code , found some typo in the code related to naming 
> variables 
> - +slowTaskRelativeTresholds+ spells wrong can be fixed to 
> +slowTaskRelativeThresholds+



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.2.4 release

[Akira Ajisaka]

Thank you Masatake for proceeding the release process.

-Akira

On Mon, Jul 11, 2022 at 10:12 AM Masatake Iwasaki 
wrote:

> I'm going to cut branch-3.2.4 today.
> If we need more time for 3.3.4 to fix the issue around HADOOP-18033,
> 3.2.4 can be released first.
>
> Thanks,
> Masatake Iwasaki
>
> On 2022/05/10 0:02, Masatake Iwasaki wrote:
> > Hi team,
> >
> > Shaded client artifacts (hadoop-client-api and hadoop-client-runtime)
> > of Hadoop 3.2.3 published to Maven turned out to be broken
> > due to issue of the release process.
> >
> > In addition, we have enough fixes on branch-3.2 after branch-3.2.3 was
> created[1].
> > Migration from log4j to reload4j is one of the major issues.
> >
> > I would like to cut RC of 3.2.4 soon after 3.3.3 release.
> > I volunteer to take a release manager role as done for 3.2.3.
> >
> > [1]
> https://issues.apache.org/jira/issues/?filter=12350757=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%203.2.4
> >
> > Thanks,
> > Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (MAPREDUCE-7383) TestLocalDistributedCacheManager fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7383:


 Summary: TestLocalDistributedCacheManager fails
 Key: MAPREDUCE-7383
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7383
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


TestLocalDistributedCacheManager#testDownload and #testDuplicateDownload is 
failing on trunk

https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/872/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt
{code:java}
[INFO] Running org.apache.hadoop.mapred.TestLocalDistributedCacheManager
[ERROR] Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.238 s 
<<< FAILURE! - in org.apache.hadoop.mapred.TestLocalDistributedCacheManager
[ERROR] 
testDuplicateDownload(org.apache.hadoop.mapred.TestLocalDistributedCacheManager)
  Time elapsed: 0.242 s  <<< ERROR!
java.io.IOException: java.util.concurrent.ExecutionException: 
java.lang.reflect.UndeclaredThrowableException
at 
org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:140)
at 
org.apache.hadoop.mapred.TestLocalDistributedCacheManager.testDuplicateDownload(TestLocalDistributedCacheManager.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.reflect.UndeclaredThrowableException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:136)
... 31 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1935)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:422)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Threa

[jira] [Resolved] (MAPREDUCE-7377) Remove unused imports in MapReduce project

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7377.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

Merged into trunk. Thank you [~groot] for the cleanup.

> Remove unused imports in MapReduce project
> --
>
> Key: MAPREDUCE-7377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7377
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> h3. Optimize Imports to keep code clean
>  # Remove any unused imports



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7380) TestLocalDistributedCacheManager is failing

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7380:


 Summary: TestLocalDistributedCacheManager is failing
 Key: MAPREDUCE-7380
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7380
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Akira Ajisaka


{code:java}
[INFO] Running org.apache.hadoop.mapred.TestLocalDistributedCacheManager
[ERROR] Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.985 s 
<<< FAILURE! - in org.apache.hadoop.mapred.TestLocalDistributedCacheManager 
{code}
[https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/866/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.3

[Akira Ajisaka]

Hi Chao,

How about using
https://repository.apache.org/content/repositories/orgapachehadoop-1348/
instead of https://repository.apache.org/content/repositories/staging/ ?

Akira

On Sat, May 7, 2022 at 10:52 AM Ayush Saxena  wrote:

> Hmm, I see the artifacts ideally should have got overwritten by the new
> RC, but they didn’t. The reason seems like the staging path shared doesn’t
> have any jars…
> That is why it was picking the old jars. I think Steve needs to run mvn
> deploy again…
>
> Sent from my iPhone
>
> > On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> >
> > 
> >>
> >> Chao can you use the one that Steve mentioned in the mail?
> >
> > Hmm how do I do that? Typically after closing the RC in nexus the
> > release bits will show up in
> >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > and Spark build will be able to pick them up for testing. However in
> > this case I don't see any 3.3.3 jars in the URL.
> >
> >> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena  wrote:
> >>
> >> There were two 3.3.3 staged. The earlier one was with skipShade, the
> date was also april 22, I archived that. Chao can you use the one that
> Steve mentioned in the mail?
> >>
> >>> On Sat, 7 May 2022 at 06:18, Chao Sun  wrote:
> >>>
> >>> Seems there are some issues with the shaded client as I was not able
> >>> to compile Apache Spark with the RC
> >>> (https://github.com/apache/spark/pull/36474). Looks like it's compiled
> >>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>> contain any class:
> >>>
> >>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>> META-INF/
> >>> META-INF/MANIFEST.MF
> >>> META-INF/NOTICE.txt
> >>> META-INF/LICENSE.txt
> >>> META-INF/maven/
> >>> META-INF/maven/org.apache.hadoop/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>
> >>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> 
>  +1 (binding)
> 
>   * Signature: ok
>   * Checksum : passed
>   * Rat check (1.8.0_191): passed
>    - mvn clean apache-rat:check
>   * Built from source (1.8.0_191): failed
>    - mvn clean install  -DskipTests
>    - mvn -fae --no-transfer-progress -DskipTests
> -Dmaven.javadoc.skip=true
>  -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
>  -Drequire.zstd -Drequire.test.libhadoop clean install
>   * Unit tests pass (1.8.0_191):
> - HDFS Tests passed (Didn't run more than this).
> 
>  Deployed a ten node ha hdfs cluster with three namenodes and five
>  journalnodes. Ran a ten node hbase (older version of 2.5 branch built
>  against 3.3.2) against it. Tried a small verification job. Good. Ran a
>  bigger job with mild chaos. All seems to be working properly
> (recoveries,
>  logs look fine). Killed a namenode. Failover worked promptly. UIs look
>  good. Poked at the hdfs cli. Seems good.
> 
>  S
> 
>  On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> 
>  wrote:
> 
> > I have put together a release candidate (rc0) for Hadoop 3.3.3
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> >
> > The git tag is release-3.3.3-RC0, commit d37586cbda3
> >
> > The maven artifacts are staged at
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> >
> > Release notes
> >
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> >
> > There's a very small number of changes, primarily critical
> code/packaging
> > issues and security fixes.
> >
> >
> >   - The critical fixes which shipped in the 3.2.3 release.
> >   -  CVEs in our code and dependencies
> >   - Shaded client packaging issues.
> >   - A switch from log4j to reload4j
> >
> >
> > reload4j is an active fork of the log4j 1.17 library with the
> classes which
> > contain CVEs removed. Even though hadoop never used those classes,
> they
> > regularly raised alerts on security scans and concen from users.
> Switching
> > to the forked project allows us to ship a secure logging framework.
> It will
> > complicate the builds of downstream maven/ivy/gradle projects which
> exclude
> > our log4j artifacts, as they need to cut the new dependency
> instead/as
> > well.
> >
> > See the release notes for details.
> >
> > This is my first release through the new docker build process, do
> please
> > validate artifact signing  to make sure it is good. I'll be trying
> builds
> > of downstream 

Re: [E] [NOTICE] Attaching patches in JIRA issue no longer works

[Akira Ajisaka]

Hi Eric,

> If we're not using patches on JIRA anymore, why are we using JIRA at all?

JIRA issues contain useful information in the fields. We are leveraging
them in development and release process.

> Using JIRA to then redirect to GitHub seems unintuitive and will fracture
the information between two different places.

Agreed that it's ideal to have all the information in one place, but the
pre commit jobs for JIRA have some limitations (
https://issues.apache.org/jira/browse/HADOOP-17798) and I don't want to
maintain the jobs anymore.

> I think this deserves some attention.

Yes, but the developers don't seem to look at the JIRA issue or read the
discussion thread. That's why I sent the [NOTICE] mail.

> we completely changed the way

Really? Most of the Hadoop developers currently use GitHub PR for code
review.

> I'm concerned that the decision was made without community
support/consensus and without a vote thread

The background is to reduce my workload for maintaining the precommit jobs
and to improve the process. I didn't think we needed a vote.
Anyway, the change is a 2-way door decision, I'm okay to revert the change
and start a discussion & vote.

-Akira

On Fri, Apr 1, 2022 at 2:02 AM Eric Badger  wrote:

> I think this deserves some attention. More than just the question of JIRA
> vs GitHub Issues, I'm a little concerned that we completely changed the way
> we post code changes without a vote thread or even a discussion thread that
> had a clear outcome. The previous thread ([DISCUSS] Tips for improving
> productivity, workflow in the Hadoop project?) had many committers giving
> opinions on the matter, but it never came to conclusion and just sat there
> with no traffic for months. The way I read the previous thread was that
> committers were proposing that we clean out stale PRs, not that we turn
> off JIRA patches/Precommit builds.
>
> I'm not necessarily saying that we should go with patches vs GitHub PRs,
> but I'm concerned that the decision was made without community
> support/consensus and without a vote thread (not sure if that's necessary
> for this type of change or not).
>
> Eric
>
> On Mon, Mar 28, 2022 at 1:18 PM Eric Badger  wrote:
>
>> If we're not using patches on JIRA anymore, why are we using JIRA at all?
>> Why don't we just use GitHub Issues? Using JIRA to then redirect to GitHub
>> seems unintuitive and will fracture the information between two different
>> places. Do the conversations happen on JIRA or on a GitHub PR? Having
>> conversations on both is confusing and splitting information. I would
>> rather use JIRA with patches or GitHub Issues with PRs. I think anything in
>> between splits information and makes it hard to find.
>>
>> Eric
>>
>> On Sun, Mar 27, 2022 at 1:25 PM Akira Ajisaka 
>> wrote:
>>
>>> Dear Hadoop developers,
>>>
>>> I've disabled the Precommit-(HADOOP|HDFS|MAPREDUCE|YARN)-Build jobs.
>>> If you attach a patch to a JIRA issue, the Jenkins precommit job won't
>>> run.
>>> Please use GitHub PR for code review.
>>>
>>> Background:
>>> -
>>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/HADOOP-17798__;!!Op6eflyXZCqGR5I!Swsnm6LmEvbzZPTXn9xJuCkXtLBzb7zHkK2P_Cw-dH5K2IwoSEzQBC2oQG0D$
>>> -
>>> https://urldefense.com/v3/__https://lists.apache.org/thread/6g3n4wo3b3tpq2qxyyth3y8m9z4mcj8p__;!!Op6eflyXZCqGR5I!Swsnm6LmEvbzZPTXn9xJuCkXtLBzb7zHkK2P_Cw-dH5K2IwoSEzQBHA0JrdK$
>>>
>>> Thanks and regards,
>>> Akira
>>>
>>

[NOTICE] Attaching patches in JIRA issue no longer works

[Akira Ajisaka]

Dear Hadoop developers,

I've disabled the Precommit-(HADOOP|HDFS|MAPREDUCE|YARN)-Build jobs.
If you attach a patch to a JIRA issue, the Jenkins precommit job won't run.
Please use GitHub PR for code review.

Background:
- https://issues.apache.org/jira/browse/HADOOP-17798
- https://lists.apache.org/thread/6g3n4wo3b3tpq2qxyyth3y8m9z4mcj8p

Thanks and regards,
Akira

Re: [VOTE] Release Apache Hadoop 3.2.3 - RC1

[Akira Ajisaka]

+1 (binding)

- Verified the signatures and the checksums
- Built from source using the "start-build-env.sh" from Ubuntu laptop.
- Setup pseudo cluster and ran some mapreduce jobs.
- Checked the Web UIs and the daemon logs.

Thanks,
Akira

On Sun, Mar 20, 2022 at 2:33 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Hi all,
>
> Here's Hadoop 3.2.3 release candidate #1:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-3.2.3-RC1/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-3.2.3-RC1
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1342
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 2.10.2 release

[Akira Ajisaka]

Thank you Masatake.

+1 to release 2.10.2.

On Fri, Mar 4, 2022 at 6:52 PM Masatake Iwasaki 
wrote:

> Hi team,
>
> There are over 170 fixed issues in branch-2.10 after the release of
> 2.10.1[1].
> Given that there is still a need for 2.10.x, I would like to release 2.10.2
> after HADOOP-18088[2] (migration to reload4j) is merged.
> I volunteer to take a release manager role as done for 2.10.1.
>
> Maybe we can declare EOL of branch-2.10 after the release,
> it should be discussed in another thread.
>
> [1]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.10.2
> [2] https://issues.apache.org/jira/browse/HADOOP-18088
>
> Thanks,
> Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

[Akira Ajisaka]

Thank you Masatake and Chao!

On Fri, Jan 28, 2022 at 5:11 PM Chao Sun  wrote:

> Thanks Masatake and Akira for discovering the issue. I used
> `dev-support/bin/create-release` which runs `mvn deploy -DskipTests
> -Pnative -Pdist ...` in a separate container and somehow it didn't hit this
> issue.
>
> Let me cherry-pick https://issues.apache.org/jira/browse/YARN-10561 to
> branch-3.3.2 and start another RC then.
>
> Thanks,
> Chao
>
> On Fri, Jan 28, 2022 at 12:01 AM Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp> wrote:
>
>> Thanks, Akira.
>>
>> I confirmed that the issue is fixed in current branch-3.3 containing
>> YARN-10561.
>>
>> On 2022/01/28 14:25, Akira Ajisaka wrote:
>> > Hi Masatake,
>> >
>> > I faced the same error in a clean environment and
>> https://issues.apache.org/jira/browse/YARN-10561 <
>> https://issues.apache.org/jira/browse/YARN-10561> should fix this issue.
>> I'll rebase the patch shortly.
>> >
>> > By the way, I'm afraid there is no active maintainer in
>> hadoop-yarn-applications-catalog module. The module is for a sample
>> application catalog, so I think we can move the module to a separate
>> repository. Of course, it should be discussed separately.
>> >
>> > Thanks and regards,
>> > Akira
>> >
>> > On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp <mailto:iwasak...@oss.nttdata.co.jp>> wrote:
>> >
>> > Thanks for putting this up, Chao Sun.
>> >
>> > I got following error on building the RC3 source tarball.
>> > It is reproducible even in the container launched by
>> `./start-build-env.sh`.
>> > There seems to be no relevant diff between release-3.3.2-RC0 and
>> release-3.3.2-RC3 (and trunk)
>> > under hadoop-yarn-applications-catalog-webapp.
>> >
>> > I guess developers having caches of related artifacts under ~/.m2
>> did not see this?
>> >
>> > ```
>> > $ mvn clean install -DskipTests -Pnative -Pdist
>> > ...
>> > [INFO] Installing node version v8.11.3
>> > [INFO] Downloading
>> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz <
>> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz> to
>> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
>> > [INFO] No proxies configured
>> > [INFO] No proxy was configured, downloading directly
>> > [INFO] Unpacking
>> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
>> into
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
>> > [INFO] Copying node binary from
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
>> to
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
>> > [INFO] Installed node locally.
>> > [INFO] Installing Yarn version v1.7.0
>> > [INFO] Downloading
>> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
>> <
>> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz>
>> to
>> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
>> > [INFO] No proxies configured
>> > [INFO] No proxy was configured, downloading directly
>> > [INFO] Unpacking
>> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
>> into
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
>> > [INFO] Installed Yarn locally.
>> > [INFO]
>> > [INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @
>> hadoop-yarn-applications-catalog-webapp ---
>> > [INFO] testFailureIgnore property is ignored in non test phases
>> > [INFO] Running 'yarn ' in
>> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-c

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

[Akira Ajisaka]

Hi Masatake,

I faced the same error in a clean environment and
https://issues.apache.org/jira/browse/YARN-10561 should fix this issue.
I'll rebase the patch shortly.

By the way, I'm afraid there is no active maintainer in
hadoop-yarn-applications-catalog module. The module is for a sample
application catalog, so I think we can move the module to a separate
repository. Of course, it should be discussed separately.

Thanks and regards,
Akira

On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> Thanks for putting this up, Chao Sun.
>
> I got following error on building the RC3 source tarball.
> It is reproducible even in the container launched by
> `./start-build-env.sh`.
> There seems to be no relevant diff between release-3.3.2-RC0 and
> release-3.3.2-RC3 (and trunk)
> under hadoop-yarn-applications-catalog-webapp.
>
> I guess developers having caches of related artifacts under ~/.m2 did not
> see this?
>
> ```
> $ mvn clean install -DskipTests -Pnative -Pdist
> ...
> [INFO] Installing node version v8.11.3
> [INFO] Downloading
> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz to
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> [INFO] No proxies configured
> [INFO] No proxy was configured, downloading directly
> [INFO] Unpacking
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> into
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
> [INFO] Copying node binary from
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
> to
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
> [INFO] Installed node locally.
> [INFO] Installing Yarn version v1.7.0
> [INFO] Downloading
> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
> to
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> [INFO] No proxies configured
> [INFO] No proxy was configured, downloading directly
> [INFO] Unpacking
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> into
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
> [INFO] Installed Yarn locally.
> [INFO]
> [INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @
> hadoop-yarn-applications-catalog-webapp ---
> [INFO] testFailureIgnore property is ignored in non test phases
> [INFO] Running 'yarn ' in
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target
> [INFO] yarn install v1.7.0
> [INFO] info No lockfile found.
> [INFO] [1/4] Resolving packages...
> [INFO] [2/4] Fetching packages...
> [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
> incompatible with this module. Expected version ">=10".
> [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
> incompatible with this module. Expected version ">=10".info Visit
> https://yarnpkg.com/en/docs/cli/install for documentation about this
> command.
> [INFO] error Found incompatible module
> [INFO]
> 
> ```
>
> Masatake Iwasaki
>
>
> On 2022/01/27 4:16, Chao Sun wrote:
> > Hi all,
> >
> > I've put together Hadoop 3.3.2 RC3 below:
> >
> > The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC3/
> > The RC tag is at:
> > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC3
> > The Maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1333
> >
> > You can find my public key at:
> > https://downloads.apache.org/hadoop/common/KEYS
> >
> > The only delta between this and RC2 is the addition of the following fix:
> >- HADOOP-18094. Disable S3A auditing by default.
> >
> > I've done the same tests as in RC2 and they look good:
> > - Ran all the unit tests
> > - Started a single node HDFS cluster and tested a few simple commands
> > - Ran all the tests in Spark using the RC2 artifacts
> >
> > Please evaluate the RC and vote, thanks!
> >
> > Best,
> > Chao
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Possibility of using ci-hadoop.a.o for Nutch integration tests

[Akira Ajisaka]

(Adding builds@)

Hi Lewis,

Nutch is already using ci-builds.apache.org, so I think Nutch can continue
using it. ci-hadoop.apache.org provides almost the same functionality as
ci-builds.apache.org and there is no non-production Hadoop cluster running
there. Therefore moving to ci-hadoop does not make sense.

Short history: In the past there were some jenkins hosts that were labeled
for Hadoop and its related projects. After the migration to cloudbees, the
labeled hosts are moved under ci-hadoop.apache.org.

Thanks,
Akira


On Thu, Jan 6, 2022 at 2:20 PM lewis john mcgibbney 
wrote:

> Thank you for the response and for directing the conversation to the
> correct places.
> I may have misunderstood what ci-hadoop.apache.org actually is. We are
> looking for a non-production Hadoop cluster which we can use to simulate
> Nutch jobs. I am not sure if this is what ci-hadoop.apache.org actually
> is...
> Instead it looks like lots of compute resources used to perform Jenkins
> CI/CD tasks for Hadoop and associated projects rather than test things
> on-top of Hadoop (and associated projects).
> Any clarity on what ci-hadoop.apache.org actually is would be greatly
> appreciated.
>
> Let me also clarify my language, rather than have the integration tests run
> on every PR, we could trigger the integration tests to be run by tagging a
> Github bot i.e., "@nutchbot integration-test". Similar to what is done with
> Dependabot or conda-forge for anyon familiar with those mechanisms.
>
> Thanks for any advice or comments.
> lewismc
>
> On Wed, Jan 5, 2022 at 9:05 PM Ayush Saxena  wrote:
>
> > Moved to Dev lists.
> >
> > Not sure about this though:
> >  when a PR is submitted to Nutch project it will run some MR job in
> Hadoop
> > CI.
> >
> > Whatever that PR requires should run as part of Nutch Infra. Why in
> Hadoop
> > CI?
> > Our CI is already loaded with our own workloads.
> > If by any chance the above assertion gets a pass, then secondly we have
> > very less number of people managing work related to CI and Infra. I don’t
> > think most of the people won’t have context or say in the Nutch project,
> > neither bandwidth to fix stuff if it gets broken.
> >
> > Just my thoughts. Looped in the dev lists, if others have any feedback.
> As
> > for the process, this would require a consensus from the Hadoop PMC
> >
> > -Ayush
> >
> > > On 06-Jan-2022, at 7:02 AM, lewis john mcgibbney 
> > wrote:
> > >
> > > Hi general@,
> > >
> > > Not sure if this is the correct mailing list. Please redirect me if
> there
> > > is a more suitable location. Thank you
> > >
> > > I am PMC over on the Nutch project (https://nutch.apache.org). I would
> > like
> > > to investigate whether we can build an integration testing capability
> for
> > > the project. This would involve running a Nutch integration test suite
> > > (collection of MR jobs) in a Hadoop CI environment. For example
> whenever
> > a
> > > pull request is submitted to the Nutch project. This could easily be
> > > automated through Jenkins.
> > >
> > > I’m not sure if this is something the Hadoop PMC would consider. Thank
> > you
> > > for the consideration.
> > >
> > > lewismc
> > > --
> > > http://home.apache.org/~lewismc/
> > > http://people.apache.org/keys/committer/lewismc
> >
>
>
> --
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

[Akira Ajisaka]

Hi Chao,

> For this, I just need to update
the hadoop-project/src/site/markdown/index.md.vm and incorporate notable
changes made in 3.3.1/3.3.2, is that correct?

> ARM binaries
There was a discussion [1] and concluded that ARM binaries are optional. The
release vote is only for source code, not binary.

BTW, I want to include https://issues.apache.org/jira/browse/YARN-11053 to
3.3.2 because it is a regression in 3.3.x releases.

[1]: https://lists.apache.org/thread/ghcgq5s745zs5cc84n4owfphf1h21zz2

Thanks and regards,
Akira



On Wed, Dec 15, 2021 at 7:56 AM Chao Sun  wrote:

> Thanks all for taking a look! looks like we need another RC addressing the
> following issues.
>
> > 1. the overview page of the doc is for the Hadoop 3.0 release. It would
> be best to base the doc on top of Hadoop 3.3.0 overview page. (it's a miss
> on my part... The overview page of 3.3.1 wasn't updated)
>
> For this, I just need to update
> the hadoop-project/src/site/markdown/index.md.vm and incorporate notable
> changes made in 3.3.1/3.3.2, is that correct? looks like the file hasn't
> been touched for a while.
>
> > 2. ARM binaries is not included. For the 3.3.1 release, I had to run the
> create release script on an ARM machine separately to create the binary
> tarball.
>
> Hmm this might be challenging for me. Could you share the steps of how you
> did it? especially where did you get an ARM machine.
>
> > 3. the jdiff version
>
> https://github.com/apache/hadoop/blob/branch-3.3.2/hadoop-project-dist/pom.xml#L137
>
> I just need to backport this commit:
>
> https://github.com/apache/hadoop/commit/a77bf7cf07189911da99e305e3b80c589edbbfb5
> to branch-3.3.2 (and potentially branch-3.3)?
>
> > The 3.3.1 binary tarball is 577mb. The 3.3.2 RC0 is 608mb. I'm curious
> what are added.
>
> The difference is mostly in aws-java-sdk-bundle jar: 3.3.1 uses
>
> https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.901
> while 3.3.2 uses
>
> https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.1026
> .
> The difference is ~32.5mb.
>
> Chao
>
> On Tue, Dec 14, 2021 at 5:25 AM Steve Loughran 
> wrote:
>
> > I'll do my best to test this; I'm a bit broken right now.
> >
> > I think we should mention in a release notes that is the version of a
> > log4j included in this and all previous releases is not vulnerable. But
> > provide a list plus links to any that have been fixed
> >
> > On Fri, 10 Dec 2021 at 02:09, Chao Sun  wrote:
> >
> >> Hi all,
> >>
> >> Sorry for the long delay. I've prepared RC0 for Hadoop 3.3.2 below:
> >>
> >> The RC is available at:
> >> http://people.apache.org/~sunchao/hadoop-3.3.2-RC0/
> >> The RC tag is at:
> >> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC0
> >> The Maven artifacts are staged at:
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1330/
> >>
> >> You can find my public key at: https://people.apache.org/~sunchao/KEYS
> >>
> >> Please evaluate the RC and vote.
> >>
> >> Thanks,
> >> Chao
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

[Akira Ajisaka]

Thank you Chao Sun for preparing the RC.
I've added your public key to the Hadoop KEYS file (
https://downloads.apache.org/hadoop/common/KEYS).

Thanks,
Akira


On Fri, Dec 10, 2021 at 11:10 AM Chao Sun  wrote:

> Hi all,
>
> Sorry for the long delay. I've prepared RC0 for Hadoop 3.3.2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC0/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC0
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1330/
>
> You can find my public key at: https://people.apache.org/~sunchao/KEYS
>
> Please evaluate the RC and vote.
>
> Thanks,
> Chao
>

Re: Hadoop-3.2.3 Release Update

[Akira Ajisaka]

Hi Brahma,

What is the release process going on? Is there any blocker for the RC?

-Akira

On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He  wrote:

> Hi Brahma,
>
> The feature 'BPServiceActor processes commands from NameNode
> asynchronously' has been ready for both branch-3.2 and branch-3.2.3. While
> cherry-picking there is only minor conflict, So I checked in directly. BTW,
> run some unit tests and build pseudo cluster to verify, it seems to work
> fine.
> FYI.
>
> Regards,
> - He Xiaoqiao
>
> On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula 
> wrote:
>
>> Please go ahead. Let me know any help required on review.
>>
>> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He  wrote:
>>
>>> Hi Brahma,
>>>
>>> I plan to involve HDFS-14997 and related JIRAs if possible. I have
>>> resolved the conflict and verified them locally.
>>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113.
>>> I would like to hear some more response that if we have enough time to
>>> wait for it to be ready.
>>> Thanks.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He  wrote:
>>>
>>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI.
>>>>
>>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula 
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly this
>>>>> can
>>>>> be done by this week,then I will try to create the RC next if there is
>>>>> no
>>>>> objection.
>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-15160
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula <
>>>>> bra...@apache.org>
>>>>> wrote:
>>>>>
>>>>> > @Akira Ajisaka   and @Masatake Iwasaki
>>>>> > 
>>>>> > Looks all are build related issues when you try with bigtop. We can
>>>>> > discuss and prioritize this.. Will connect with you guys.
>>>>> >
>>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki <
>>>>> > iwasak...@oss.nttdata.co.jp> wrote:
>>>>> >
>>>>> >> >> -
>>>>> >>
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>>>> >> >
>>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>>>> >> > we do not see the issue usually since branch-3.2 uses
>>>>> zooekeper-3.4.13,
>>>>> >> > while it would be harmless to add the exclusion even for
>>>>> >> zooekeeper-3.4.13.
>>>>> >>
>>>>> >> I filed HADOOP-17849 for this.
>>>>> >>
>>>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote:
>>>>> >> > Thanks for bringing this up, Akira. Let me explain some
>>>>> background.
>>>>> >> >
>>>>> >> >
>>>>> >> >> -
>>>>> >>
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
>>>>> >> >
>>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14.
>>>>> >> > we do not see the issue usually since branch-3.2 uses
>>>>> zooekeper-3.4.13,
>>>>> >> > while it would be harmless to add the exclusion even for
>>>>> >> zooekeeper-3.4.13.
>>>>> >> >
>>>>> >> >
>>>>> >> >> -
>>>>> >>
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
>>>>> >> >> -
>>>>> >>
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
>>>>> >> >> -
>>>>> >>
>>>>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
>>>>> >> >
>>>&

Re: [DISCUSS] Migrate to Yetus Interface classification annotations

[Akira Ajisaka]

Hi Masatake,

The problem comes from the removal of com.sun.tools.doclets.* packages in
Java 10.
In Apache Hadoop, I removed the doclet support for filtering javadocs when
the environment is Java 10 or upper.
https://issues.apache.org/jira/browse/HADOOP-15304

Thanks,
Akira

On Tue, Sep 28, 2021 at 10:27 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> > In particular, there has been an outstanding problem with doclet support
> for filtering javadocs by annotation since JDK9 came out.
>
> Could you give me a pointer to relevant Yetus JIRA or ML thread?
>
> On 2021/09/28 1:17, Sean Busbey wrote:
> > I think consolidating on a common library and tooling for defining API
> expectations for Hadoop would be great.
> >
> > Unfortunately, the Apache Yetus community recently started a discussion
> around dropping their maintenance of the audience annotations codebase[1]
> due to lack of community interest. In particular, there has been an
> outstanding problem with doclet support for filtering javadocs by
> annotation since JDK9 came out.
> >
> > I think that means a necessary first step here would be to determine if
> we have contributors willing to show up over in that project to get things
> into a good state for future JDK adoption.
> >
> >
> >
> > [1]:
> > https://s.apache.org/ybdl6
> > "[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org
> >
> >> On Sep 27, 2021, at 2:46 AM, Viraj Jasani  wrote:
> >>
> >> Since the early days, Hadoop has provided Interface classification
> >> annotations to represent the scope and stability for downstream
> >> applications to select Hadoop APIs carefully. After some time, these
> >> annotations (InterfaceAudience and InterfaceStability) have been
> migrated
> >> to Apache Yetus. As of today, with increasing number of Hadoop ecosystem
> >> applications using (or starting to use) Yetus stability annotations for
> >> their own downstreamers, we should also consider using IA/IS annotations
> >> provided by *org.apache.yetus.audience *directly in our codebase and
> retire
> >> our *org.apache.hadoop.classification* package for the better
> separation of
> >> concern and single source.
> >>
> >> I believe we can go with this migration to maintain compatibility for
> >> Hadoop downstreamers:
> >>
> >>1. In Hadoop trunk (3.4.0+ releases), replace all usages of o.a.h.c
> >>stability annotations with o.a.y.a annotations.
> >>2. Deprecate o.a.h.c annotations, and provide deprecation warning
> that
> >>we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only
> source for
> >>these annotations should be o.a.y.a.
> >>
> >> Any thoughts?
> >
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Hadoop-3.2.3 Release Update

[Akira Ajisaka]

Thanks Brahma for cutting branch-3.2.3.

In Apache Bigtop, there are some patches applied to Hadoop 3.2.2.
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop

In these patches, how about backporting the following issues to
branch-3.2 and branch-3.2.3?
- HADOOP-14922
- HADOOP-15939
- HADOOP-17569

In addition, there are some patches that don't have JIRA issue ID.
Maybe we need to create JIRAs and fix those.
- 
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff
- 
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff
- 
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff
- 
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff
- 
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff

Thanks and regards,
Akira

On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He  wrote:
>
> Thanks Brahma for initiating this and making hadoop-3.2.3 release happen.
>
> I would like to validate the HBase project (both the latest release and
> trunk branch).
> Chao Sun will validate the Spark Project (Got in touch with Chao already).
> once RC is out.
>
> Thanks and Regards,
> - He Xiaoqiao
>
>
> On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula 
> wrote:
>
> > Hi All,
> >
> > I cut branch-3.2.3 and it is ready for release. Please commit to
> > branch-3.2.3 if any critical/blocker issues need to go.
> >
> > *This time I am thinking of having downstream projects and companies'
> > voices,let's know how this can go. *
> >
> >- Planning to check with downstream projects like Spark,HBase and Hive
> >if they can help on validation(Or running their UT on this branch)
> >- Collecting info from Companies who are already deployed and using the
> >branch-3.2
> >
> >
> > so that we can make a more stable release on 3.2 (so that features released
> > on this branch impact can be known) ,please let me know anybody from these
> > communities who can help on this.
> >
> >
> > Planning to create RC by this month end. Any suggestions are welcome.
> >
> >
> >
> > --Brahma Reddy Battula
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.2.3 release

[Akira Ajisaka]

Hi Steven,

Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
opened a PR to backport YARN-8990:
https://github.com/apache/hadoop/pull/3254

Thanks,
Akira

On Thu, Jul 29, 2021 at 10:36 AM Steven Rand  wrote:
>
> I think it would be helpful if we could include YARN-8990 and YARN-8992 in 
> the 3.2.3 release. Both are important fixes which were included in 3.2.0, but 
> never made their way to branch-3.2, so were omitted from both 3.2.1 and 3.2.2.
>
> Best,
> Steve
>
> On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He  wrote:
>>
>> cc @dev mail-list.
>>
>> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He  wrote:
>>
>> > Hi Brahma,
>> >
>> > I just created version 3.2.4, and changed all unresolved issues (target
>> > version/s: 3.2.3) to 3.2.4 after checking both of them are not blocker
>> > issues. Dashboard[1] is clean now.
>> >
>> > Regards,
>> > - He Xiaoqiao
>> >
>> > [1]
>> > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
>> >
>> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula 
>> > wrote:
>> >
>> >> Hi Xiaoqiao,
>> >>
>> >> Thanks for creating the Dashboard, we need to change the filters and
>> >> target versions in the jira.
>> >>
>> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He  wrote:
>> >>
>> >>> Thanks Brahma for volunteering and driving this release plan. I just
>> >>> created a dashboard for 3.2.3 release[1].
>> >>> I would like to support for this release line if need. (cc Brahma)
>> >>>
>> >>> Thanks. Regards,
>> >>> - He Xiaoqiao
>> >>>
>> >>> [1]
>> >>>
>> >>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
>> >>>
>> >>>
>> >>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka 
>> >>> wrote:
>> >>>
>> >>> > Hi Brahma,
>> >>> >
>> >>> > Thank you for volunteering!
>> >>> >
>> >>> > -Akira
>> >>> >
>> >>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula <
>> >>> bra...@apache.org>
>> >>> > wrote:
>> >>> > >
>> >>> > > Hi Akira,
>> >>> > >
>> >>> > > Thanks for bringing this..
>> >>> > >
>> >>> > > I want to drive this if nobody already plan to do this..
>> >>> > >
>> >>> > >
>> >>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka 
>> >>> > wrote:
>> >>> > >
>> >>> > > > Hi all,
>> >>> > > >
>> >>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
>> >>> > > > accumulated more than 230 commits [1]. Therefore I want to start
>> >>> the
>> >>> > > > release work for 3.2.3.
>> >>> > > >
>> >>> > > > There is one blocker for 3.2.3 [2].
>> >>> > > > - https://issues.apache.org/jira/browse/HDFS-12920
>> >>> > > >
>> >>> > > > Is there anyone who would volunteer to be the 3.2.3 release
>> >>> manager?
>> >>> > > > Are there any other blockers? If any, please file an issue, raise
>> >>> the
>> >>> > > > blocker, and add the target version.
>> >>> > > >
>> >>> > > > [1]
>> >>> > > >
>> >>> >
>> >>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
>> >>> > > > [2]
>> >>> > > >
>> >>> >
>> >>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
>> >>> > > >
>> >>> > > > Regards,
>> >>> > > > Akira
>> >>> > > >
>> >>> > > >
>> >>> -
>> >>> > > > To unsubscribe, e-mail:
>> >>> mapreduce-dev-unsubscr...@hadoop.apache.org
>> >>> > > > For additional commands, e-mail:
>> >>> mapreduce-dev-h...@hadoop.apache.org
>> >>> > > >
>> >>> > > > --
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > --Brahma Reddy Battula
>> >>> >
>> >>> > -
>> >>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >>
>> >>
>> >> --Brahma Reddy Battula
>> >>
>> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7258) HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from "taskcounterGroup" to "taskCounterGroup".

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7258.
--
Fix Version/s: 3.3.2
   3.2.3
   3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged the PR into trunk, branch-3.3, and branch-3.2.

> HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's 
> itemName from "taskcounterGroup" to "taskCounterGroup".
> --
>
> Key: MAPREDUCE-7258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7258
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.1.2, 3.1.3
>Reporter: jenny
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Attachments: image-2020-01-16-14-05-30-007.png, 
> image-2020-01-16-14-32-02-183.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's 
> itemName from taskcounterGroup to taskCounterGroup.
>  !image-2020-01-16-14-05-30-007.png|width=452,height=251!
> !image-2020-01-16-14-32-02-183.png|width=447,height=254!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.2.3 release

[Akira Ajisaka]

Hi Brahma,

Thank you for volunteering!

-Akira

On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula  wrote:
>
> Hi Akira,
>
> Thanks for bringing this..
>
> I want to drive this if nobody already plan to do this..
>
>
> On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka  wrote:
>
> > Hi all,
> >
> > Hadoop 3.2.2 was released half a year ago, and now, we have
> > accumulated more than 230 commits [1]. Therefore I want to start the
> > release work for 3.2.3.
> >
> > There is one blocker for 3.2.3 [2].
> > - https://issues.apache.org/jira/browse/HDFS-12920
> >
> > Is there anyone who would volunteer to be the 3.2.3 release manager?
> > Are there any other blockers? If any, please file an issue, raise the
> > blocker, and add the target version.
> >
> > [1]
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> > [2]
> > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
> >
> > Regards,
> > Akira
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> > --
>
>
>
> --Brahma Reddy Battula

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] Hadoop 3.2.3 release

[Akira Ajisaka]

Hi all,

Hadoop 3.2.2 was released half a year ago, and now, we have
accumulated more than 230 commits [1]. Therefore I want to start the
release work for 3.2.3.

There is one blocker for 3.2.3 [2].
- https://issues.apache.org/jira/browse/HDFS-12920

Is there anyone who would volunteer to be the 3.2.3 release manager?
Are there any other blockers? If any, please file an issue, raise the
blocker, and add the target version.

[1] 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
[2] 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3

Regards,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Akira Ajisaka]

Based on the positive feedback, filed
https://issues.apache.org/jira/browse/HADOOP-17813 to update the
checkstyle rule.
I don't think it requires a vote thread.

Thanks and regards,
Akira

On Sun, Jun 13, 2021 at 1:14 AM Steve Loughran
 wrote:
>
> +1
>
> if you look closely the hadoop-azure module went to 100 lines a while back
> and all is good
>
> On Wed, 19 May 2021 at 22:13, Sean Busbey  wrote:
>
> > Hello!
> >
> > What do folks think about changing our line length guidelines to allow for
> > 100 character width?
> >
> > Currently, we tell folks to follow the sun style guide with some exception
> > unrelated to line length. That guide says width of 80 is the standard and
> > our current check style rules act as enforcement.
> >
> > Looking at the current trunk codebase our nightly build shows a total of
> > ~15k line length violations; it’s about 18% of identified checkstyle issues.
> >
> > The vast majority of those line length violations are <= 100 characters
> > long. 100 characters happens to be the length for the Google Java Style
> > Guide, another commonly adopted style guide for java projects, so I suspect
> > these longer lines leaking past the checkstyle precommit warning might be a
> > reflection of committers working across multiple java codebases.
> >
> > I don’t feel strongly about lines being longer, but I would like to move
> > towards more consistent style enforcement as a project. Updating our
> > project guidance to allow for 100 character lines would reduce the
> > likelihood that folks bringing in new contributions need a precommit test
> > cycle to get the formatting correct.
> >
> > Does anyone feel strongly about keeping the line length limit at 80
> > characters?
> >
> > Does anyone feel strongly about contributions coming in that clear up line
> > length violations?
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-7228) Refactor ActionExecutorTestCase to not extend HCatTestCase

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reopened MAPREDUCE-7228:
--

> Refactor ActionExecutorTestCase to not extend HCatTestCase
> --
>
> Key: MAPREDUCE-7228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7228
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: test
>Reporter: Kinga Marton
>Priority: Major
>
> {{ActionExecutorTestCase}} extends {{HCatTestCase}} but there are only two 
> test cases where we need HCatalog related settings as well. 
> Let's try to refactor this part and make the ActionExecturTestCase 
> independent from the HCatalog part.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7321.
--
Resolution: Duplicate

> TestMRIntermediateDataEncryption does not cleanup data folders
> --
>
> Key: MAPREDUCE-7321
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> The data generated by the {{TestMRIntermediateDataEncryption}} does not get 
> deleted after Completing the tests. This contributes to Hadoop taking large 
> disk spaces to build and run tests.
>  The following folders need to be removed:
>  * folders of the DFSCluster and the YarnCluster
>  * Files used to submit jobs in the test-dir folder



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7228) Refactor ActionExecutorTestCase to not extend HCatTestCase

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7228.
--
Resolution: Invalid

> Refactor ActionExecutorTestCase to not extend HCatTestCase
> --
>
> Key: MAPREDUCE-7228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7228
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: test
>Reporter: Kinga Marton
>Priority: Major
>
> {{ActionExecutorTestCase}} extends {{HCatTestCase}} but there are only two 
> test cases where we need HCatalog related settings as well. 
> Let's try to refactor this part and make the ActionExecturTestCase 
> independent from the HCatalog part.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reopened MAPREDUCE-7321:
--

> TestMRIntermediateDataEncryption does not cleanup data folders
> --
>
> Key: MAPREDUCE-7321
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
>
> The data generated by the {{TestMRIntermediateDataEncryption}} does not get 
> deleted after Completing the tests. This contributes to Hadoop taking large 
> disk spaces to build and run tests.
>  The following folders need to be removed:
>  * folders of the DFSCluster and the YarnCluster
>  * Files used to submit jobs in the test-dir folder



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Hadoop 3.1.x EOL

[Akira Ajisaka]

This vote has passed with 18 binding +1. I'll update the JIRA and the wiki.

Thanks all for your participation.

On Tue, Jun 8, 2021 at 3:03 AM Steve Loughran  wrote:
>
>
>
> On Thu, 3 Jun 2021 at 07:14, Akira Ajisaka  wrote:
>>
>> Dear Hadoop developers,
>>
>> Given the feedback from the discussion thread [1], I'd like to start
>> an official vote
>> thread for the community to vote and start the 3.1 EOL process.
>>
>> What this entails:
>>
>> (1) an official announcement that no further regular Hadoop 3.1.x releases
>> will be made after 3.1.4.
>> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
>>
>> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
>>
>> Committers are eligible to cast binding votes. Non-committers are welcomed
>> to cast non-binding votes.
>>
>> Here is my vote, +1
>
>
>
> +1 (binding)
>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[VOTE] Hadoop 3.1.x EOL

[Akira Ajisaka]

Dear Hadoop developers,

Given the feedback from the discussion thread [1], I'd like to start
an official vote
thread for the community to vote and start the 3.1 EOL process.

What this entails:

(1) an official announcement that no further regular Hadoop 3.1.x releases
will be made after 3.1.4.
(2) resolve JIRAs that specifically target 3.1.5 as won't fix.

This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].

Committers are eligible to cast binding votes. Non-committers are welcomed
to cast non-binding votes.

Here is my vote, +1

[1] https://s.apache.org/w9ilb
[2] 
https://www.timeanddate.com/worldclock/fixedtime.html?msg=4=20210610T16=248

Regards,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] which release lines should we still consider actively maintained?

[Akira Ajisaka]

Thank you for your comments. I'll create a vote thread to mark 3.1.x EOL.

-Akira

On Tue, May 25, 2021 at 12:46 AM Ayush Saxena  wrote:
>
> +1, to mark 3.1.x EOL.
> Apache Hive does depends on 3.1.0 as of now, but due to guave upgrade on 
> branch-3.1, the attempt to migrate to latest 3.1.x didn’t work for me atleast 
> couple of months back. So, mostly 3.3.1 would be the only option replacing 
> 3.1.0 there or at worst 3.3.2 in a couple of months.
>
>
> -Ayush
>
> > On 24-May-2021, at 8:43 PM, Arpit Agarwal  
> > wrote:
> >
> > +1 to EOL 3.1.x at least.
> >
> >
> >> On May 23, 2021, at 9:51 PM, Wei-Chiu Chuang 
> >>  wrote:
> >>
> >> Sean,
> >>
> >> For reasons I don't understand, I never received emails from your new
> >> address in the mailing list. Only Akira's response.
> >>
> >> I was just able to start a thread like this.
> >>
> >> I am +1 to EOL 3.1.5.
> >> Reason? Spark is already on Hadoop 3.2. Hive and Tez are actively working
> >> to support Hadoop 3.3. HBase supports Hadoop 3.3 already. They are the most
> >> common Hadoop applications so I think a 3.1 isn't that necessarily
> >> important.
> >>
> >> With Hadoop 3.3.1, we have a number of improvements to support a better
> >> HDFS upgrade experience, so upgrading from Hadoop 3.1 should be relatively
> >> easy. Application upgrade takes some effort though (commons-lang ->
> >> commons-lang3 migration for example)
> >> I've been maintaining the HDFS code in branch-3.1, so from a
> >> HDFS perspective the branch is always in a ready to release state.
> >>
> >> The Hadoop 3.1 line is more than 3 years old. Maintaining this branch is
> >> getting trickier. I am +100 to reduce the number of actively maintained
> >> release line. IMO, 2 Hadoop 3 lines + 1 Hadoop 2 line is a good idea.
> >>
> >>
> >>
> >> For Hadoop 3.3 line: If no one beats me, I plan to make a 3.3.2 in 2-3
> >> months. And another one in another 2-3 months.
> >> The Hadoop 3.3.1 has nearly 700 commits not in 3.3.0. It is very difficult
> >> to make/validate a maint release with such a big divergence in the code.
> >>
> >>
> >>> On Mon, May 24, 2021 at 12:06 PM Akira Ajisaka  >>> <mailto:aajis...@apache.org>> wrote:
> >>>
> >>> Hi Sean,
> >>>
> >>> Thank you for starting the discussion.
> >>>
> >>> I think branch-2.10, branch-3.1, branch-3.2, branch-3.3, and trunk
> >>> (3.4.x) are actively maintained.
> >>>
> >>> The next releases will be:
> >>> - 3.4.0
> >>> - 3.3.1 (Thanks, Wei-Chiu!)
> >>> - 3.2.3
> >>> - 3.1.5
> >>> - 2.10.2
> >>>
> >>>> Are there folks willing to go through being release managers to get more
> >>> of these release lines on a steady cadence?
> >>>
> >>> Now I'm interested in becoming a release manager of 3.1.5.
> >>>
> >>>> If I were to take up maintenance release for one of them which should it
> >>> be?
> >>>
> >>> 3.2.3 or 2.10.2 seems to be a good choice.
> >>>
> >>>> Should we declare to our downstream users that some of these lines
> >>> aren’t going to get more releases?
> >>>
> >>> Now I think we don't need to declare that. I believe 3.3.1, 3.2.3,
> >>> 3.1.5, and 2.10.2 will be released in the near future.
> >>> There are some earlier discussions of 3.1.x EoL, so 3.1.5 may be a
> >>> final release of the 3.1.x release line.
> >>>
> >>>> Is there downstream facing documentation somewhere that I missed for
> >>> setting expectations about our release cadence and actively maintained
> >>> branches?
> >>>
> >>> As you commented, the confluence wiki pages for Hadoop releases were
> >>> out of date. Updated [1].
> >>>
> >>>> Do we have a backlog of work written up that could make the release
> >>> process easier for our release managers?
> >>>
> >>> The release process is documented and maintained:
> >>> https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
> >>> Also, there are some backlogs [1], [2].
> >>>
> >>> [1]:
> >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
> >>> [2]: h

Re: [VOTE] Release Apache Hadoop Thirdparty 1.1.1 RC0

[Akira Ajisaka]

+1

- Verified checksums and signatures
- Built from source with -Psrc profile
- Checked the documents
- Compiled Hadoop trunk and branch-3.3 with Hadoop third-party 1.1.1.

-Akira

On Wed, May 26, 2021 at 5:29 PM Wei-Chiu Chuang  wrote:
>
> Hi folks,
>
> I have put together a release candidate (RC0) for Hadoop Thirdparty
> 1.1.1 which will be consumed by Hadoop 3.3.1 RC2.
>
>
> The RC is available at:
> https://people.apache.org/~weichiu/hadoop-thirdparty-1.1.1-RC0/
>
>
> The RC tag in svn is
> here:https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.1.1-RC0
>
> The maven artifacts are staged at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1316/
>
>
> Comparing to 1.1.0, there are two additional fixes:
>
> HADOOP-17707. Remove jaeger document from site index.
> 
>
> HADOOP-17730. Add back error_prone
> 
>
> You can find my public key
> at:https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
>
> Please try the release and vote. The vote will run for 5 days.
>
> Thanks
> Weichiu

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7348) TestFrameworkUploader#testNativeIO fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7348:


 Summary: TestFrameworkUploader#testNativeIO fails
 Key: MAPREDUCE-7348
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7348
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka
 Attachments: 
patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-uploader.txt

TestFrameworkUploader#testNativeIO fails.
{noformat}
[INFO] Running org.apache.hadoop.mapred.uploader.TestFrameworkUploader
[ERROR] Tests run: 16, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 8.038 
s <<< FAILURE! - in org.apache.hadoop.mapred.uploader.TestFrameworkUploader
[ERROR] testNativeIO(org.apache.hadoop.mapred.uploader.TestFrameworkUploader)  
Time elapsed: 0.026 s  <<< ERROR!
org.apache.commons.io.IOExceptionList: 2 exceptions: [java.io.IOException: 
Unable to delete file: 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader/target/test-dir/3575932722382085229/symlinkToTarget2.txt,
 java.io.IOException: Unable to delete file: 
/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-uploader/target/test-dir/3575932722382085229/symlinkToTarget.txt]
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:345)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1206)
at 
org.apache.hadoop.mapred.uploader.TestFrameworkUploader.testNativeIO(TestFrameworkUploader.java:480)
{noformat}
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/518/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-uploader.txt




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] which release lines should we still consider actively maintained?

[Akira Ajisaka]

Hi Sean,

Thank you for starting the discussion.

I think branch-2.10, branch-3.1, branch-3.2, branch-3.3, and trunk
(3.4.x) are actively maintained.

The next releases will be:
- 3.4.0
- 3.3.1 (Thanks, Wei-Chiu!)
- 3.2.3
- 3.1.5
- 2.10.2

> Are there folks willing to go through being release managers to get more of 
> these release lines on a steady cadence?

Now I'm interested in becoming a release manager of 3.1.5.

> If I were to take up maintenance release for one of them which should it be?

3.2.3 or 2.10.2 seems to be a good choice.

> Should we declare to our downstream users that some of these lines aren’t 
> going to get more releases?

Now I think we don't need to declare that. I believe 3.3.1, 3.2.3,
3.1.5, and 2.10.2 will be released in the near future.
There are some earlier discussions of 3.1.x EoL, so 3.1.5 may be a
final release of the 3.1.x release line.

> Is there downstream facing documentation somewhere that I missed for setting 
> expectations about our release cadence and actively maintained branches?

As you commented, the confluence wiki pages for Hadoop releases were
out of date. Updated [1].

> Do we have a backlog of work written up that could make the release process 
> easier for our release managers?

The release process is documented and maintained:
https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
Also, there are some backlogs [1], [2].

[1]: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
[2]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

Thanks,
Akira

On Fri, May 21, 2021 at 7:12 AM Sean Busbey  wrote:
>
>
> Hi folks!
>
> Which release lines do we as a community still consider actively maintained?
>
> I found an earlier discussion[1] where we had consensus to consider branches 
> that don’t get maintenance releases on a regular basis end-of-life for 
> practical purposes. The result of that discussion was written up in our wiki 
> docs in the “EOL Release Branches” page, summarized here
>
> >  If no volunteer to do a maintenance release in a short to mid-term (like 3 
> > months to 1 or 1.5 year).
>
> Looking at release lines that are still on our download page[3]:
>
> * Hadoop 2.10.z - last release 8 months ago
> * Hadoop 3.1.z - last release 9.5 months ago
> * Hadoop 3.2.z - last release 4.5 months ago
> * Hadoop 3.3.z - last release 10 months ago
>
> And then trunk holds 3.4 which hasn’t had a release since the branch-3.3 fork 
> ~14 months ago.
>
> I can see that Wei-Chiu has been actively working on getting the 3.3.1 
> release out[4] (thanks Wei-Chiu!) but I do not see anything similar for the 
> other release lines.
>
> We also have pages on the wiki for our project roadmap of release[5], but it 
> seems out of date since it lists in progress releases that have happened or 
> branches we have announced as end of life, i.e. 2.8.
>
> We also have a group of pages (sorry, I’m not sure what the confluence jargon 
> is for this) for “hadoop active release lines”[6] but this list has 2.8, 2.9, 
> 3.0, 3.1, and 3.3. So several declared end of life lines and no 2.10 or 3.2 
> despite those being our release lines with the most recent releases.
>
> Are there folks willing to go through being release managers to get more of 
> these release lines on a steady cadence?
>
> If I were to take up maintenance release for one of them which should it be?
>
> Should we declare to our downstream users that some of these lines aren’t 
> going to get more releases?
>
> Is there downstream facing documentation somewhere that I missed for setting 
> expectations about our release cadence and actively maintained branches?
>
> Do we have a backlog of work written up that could make the release process 
> easier for our release managers?
>
>
> [1]: https://s.apache.org/7c8jt
> [2]: https://s.apache.org/4no96
> [3]: https://hadoop.apache.org/releases.html
> [4]: https://s.apache.org/1bvwe
> [5]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
> [6]: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Akira Ajisaka]

I'm +1 to allow <= 100 chars.

FYI: There were some discussions long before:
- 
https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
- 
https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E

Thanks,
Akira

On Thu, May 20, 2021 at 6:36 AM Sean Busbey  wrote:
>
> Hello!
>
> What do folks think about changing our line length guidelines to allow for 
> 100 character width?
>
> Currently, we tell folks to follow the sun style guide with some exception 
> unrelated to line length. That guide says width of 80 is the standard and our 
> current check style rules act as enforcement.
>
> Looking at the current trunk codebase our nightly build shows a total of ~15k 
> line length violations; it’s about 18% of identified checkstyle issues.
>
> The vast majority of those line length violations are <= 100 characters long. 
> 100 characters happens to be the length for the Google Java Style Guide, 
> another commonly adopted style guide for java projects, so I suspect these 
> longer lines leaking past the checkstyle precommit warning might be a 
> reflection of committers working across multiple java codebases.
>
> I don’t feel strongly about lines being longer, but I would like to move 
> towards more consistent style enforcement as a project. Updating our project 
> guidance to allow for 100 character lines would reduce the likelihood that 
> folks bringing in new contributions need a precommit test cycle to get the 
> formatting correct.
>
> Does anyone feel strongly about keeping the line length limit at 80 
> characters?
>
> Does anyone feel strongly about contributions coming in that clear up line 
> length violations?
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] hadoop-thirdparty 1.1.0-RC0

[Akira Ajisaka]

+1

- Verified checksums and signatures
- Built from source with -Psrc profile
- Checked the documents

-Akira

On Thu, May 13, 2021 at 8:55 PM Wei-Chiu Chuang  wrote:
>
> Hello my fellow Hadoop developers,
>
> I am putting together the first release candidate (RC0) for
> Hadoop-thirdparty 1.1.0. This is going to be consumed by the upcoming
> Hadoop 3.3.1 release.
>
> The RC is available at:
> https://people.apache.org/~weichiu/hadoop-thirdparty-1.1.0-RC0/
> The RC tag in github is here:
> https://github.com/apache/hadoop-thirdparty/tree/release-1.1.0-RC0
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1309/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> https://people.apache.org/keys/committer/weichiu.asc
>
>
> Please try the release and vote. The vote will run for 5 days until
> 2021/05/19 at 00:00 CST.
>
> Note: Our post commit automation builds the code, and pushes the SNAPSHOT
> artifacts to central Maven, which is consumed by Hadoop trunk and
> branch-3.3, so it is a good validation that things are working properly in
> hadoop-thirdparty.
>
> Thanks,
> Wei-Chiu

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7334) TestJobEndNotifier fails

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7334.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

Thank you [~jingzhao]!

> TestJobEndNotifier fails
> 
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>    Reporter: Akira Ajisaka
>    Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
> Time elapsed: 47.322 s  <<< ERROR!
> java.util.concurrent.TimeoutException
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
>   at 
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
>   at org.eclipse.jetty.server.Server.doStop(Server.java:447)
>   at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
>   at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
>   at 
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7329) HadoopPipes task may fail when linux kernel version change from 3.x to 4.x

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7329.
--
Fix Version/s: 3.3.1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to trunk and branch-3.3. Thank you [~lichaojacobs] for your 
contribution!

> HadoopPipes task may fail when linux kernel version change from 3.x to 4.x
> --
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: pipes
>Affects Versions: 2.6.0
>Reporter: chaoli
>Assignee: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 3.4.0, 3.3.1
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7335) TestMRJobClient fails by FileAlreadyExistsException

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7335:


 Summary: TestMRJobClient fails by FileAlreadyExistsException
 Key: MAPREDUCE-7335
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7335
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/10/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
{quote}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 189.115 
s <<< FAILURE! - in org.apache.hadoop.mapreduce.TestMRJobClient
[ERROR] testJobClient(org.apache.hadoop.mapreduce.TestMRJobClient)  Time 
elapsed: 77.154 s  <<< ERROR!
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory 
hdfs://localhost:39225/user/jenkins/target/output already exists
at 
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:164)
at 
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:277)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1571)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1568)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1900)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1568)
at 
org.apache.hadoop.mapreduce.TestMRJobClient.runJobInBackGround(TestMRJobClient.java:90)
at 
org.apache.hadoop.mapreduce.TestMRJobClient.testKillJob(TestMRJobClient.java:222)
at 
org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7334) TestJobEndNotifier fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7334:


 Summary: TestJobEndNotifier fails
 Key: MAPREDUCE-7334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
{quote}
[INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 s 
<<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
[ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
Time elapsed: 47.322 s  <<< ERROR!
java.util.concurrent.TimeoutException
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
at 
org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
at org.eclipse.jetty.server.Server.doStop(Server.java:447)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
at 
org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{quote}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7323) Support Python 3 in job_history_summary.py

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7323:


 Summary: Support Python 3 in job_history_summary.py
 Key: MAPREDUCE-7323
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7323
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Akira Ajisaka


Fix the following syntax error in Python 3:
{noformat}
aajisaka@b23b6a126ee7:~/hadoop$ python3 
hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py
  File 
"hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py",
 line 73
print "Name reduce-output-bytes shuffle-finish reduce-finish"
  ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Name 
reduce-output-bytes shuffle-finish reduce-finish")?
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Easy way to run all the unit tests (apache id required)

[Akira Ajisaka]

Hi folks,

I've prepared a Jenkins on-demand qbt job:
https://ci-hadoop.apache.org/job/hadoop-qbt-any-java8-linux-x86_64

If you are tired of touching the source code in the modules to run all
the tests (and seeing some broken Jenkins job results), please use
this job instead.

How to use:
- Login https://ci-hadoop.apache.org/ (apache id required)
- Before running the job, click 'Configure' and set the repository to
the PR author's repository
- Set the author's branch and e-mail recipients when running the job

If you don't have an apache id, please ping me.

Best,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC5

[Akira Ajisaka]

+1 (binding)

- Verified the checksums and signatures
- Verified the source is the same as the RC5 tag
- Built from source with CentOS 7.9 and Java 1.8.0_275
- Setup pseudo cluster and ran some FsShell commands

Thanks,
Akira

On Sun, Jan 3, 2021 at 11:40 PM Xiaoqiao He  wrote:
>
> Hi folks,
>
> The release candidate (RC5) for Hadoop-3.2.2 is available now.
> There are 4 commits[1] differences between RC5 and RC4[2].
>
> The RC5 is available at:
> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC5
> The RC5 tag in github is here:
> https://github.com/apache/hadoop/tree/release-3.2.2-RC5
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1298
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>
> Please try the release and vote.
>
> I have done a simple test.
> * Verify gpg sign and md5sum.
> * Check staging repositories point to the correct artifact(
> https://repository.apache.org/content/repositories/staging/).
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/mv/rm.
> * Submit example mr applications and check the result - Pi & wordcount.
> * Check the web UI and the release year of
> NameNode/DataNode/Resourcemanager/NodeManager including YARN UI2.
>
> My +1 to start.
>
> Thanks and Happy New Year!
> - He Xiaoqiao
>
> [1]
> https://github.com/apache/hadoop/compare/release-3.2.2-RC4...release-3.2.2-RC5
> [2]
> https://lists.apache.org/thread.html/r8911d2934ebfe3869874842be94d1cfb00d99334bc93819d71466243%40%3Chdfs-dev.hadoop.apache.org%3E
> [3]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Akira Ajisaka]

+1 (binding)

- Verified checksums and signatures
- Built from source with CentOS 7.9
- Setup a pseudo cluster and ran some MapReduce jobs
- Checked the Web UIs.

Thanks,
Akira

On Tue, Dec 15, 2020 at 6:43 AM Wei-Chiu Chuang  wrote:
>
> Diff between RC3 and RC4:
>
>  MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136)
> HDFS-13404. Addendum: RBF:
> TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail.
> Contributed by Takanobu Asanuma.
> HADOOP-16080. hadoop-aws does not work with hadoop-client-api  (#2510).
> Contributed by Chao Sun
> HADOOP-15775. [JDK9] Add missing javax.activation-api dependency.
> Contributed by Akira Ajisaka.
> HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error.
> Contributed by HuangTao.
> HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in
> branch-3.3 and branch-3.2 (#2517)
> HADOOP-17389. KMS should log full UGI principal. (#2476)
> HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by
> Ahmed Hussein and Daryn Sharp
> HDFS-15709. Socket file descriptor leak in
> StripedBlockChecksumReconstructor. (#2518)
>
>
> On Wed, Dec 9, 2020 at 9:01 AM Xiaoqiao He  wrote:
>
> > Hi folks,
> >
> > The release candidate (RC4) for Hadoop-3.2.2 is available now.
> > There are 10 commits[1] differences between RC4 and RC3[2].
> >
> > The RC4 is available at:
> > http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
> > The RC4 tag in github is here:
> > https://github.com/apache/hadoop/tree/release-3.2.2-RC4
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1296
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> > https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> >
> > Please try the release and vote.
> >
> > I have done a simple testing with my pseudo cluster.
> > * Verify gpg sign and md5sum.
> > * Setup pseudo cluster with HDFS and YARN.
> > * Run simple FsShell - mkdir/put/get/rename/mv.
> > * Submit example mr job and check the result - Pi/wordcount.
> > * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
> > My +1 to start.
> >
> > Thanks,
> > He Xiaoqiao
> >
> > [1]
> >
> > https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
> > [2]
> >
> > https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
> > [3]
> > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] Upgrading Jersey from 1.x

[Akira Ajisaka]

Hi folks,

Recently I resumed the work for upgrading Jersey:
- https://issues.apache.org/jira/browse/HADOOP-15984
- https://github.com/apache/hadoop/pull/763

This upgrade is important because we cannot compile and run Jersey 1.x
with Java 11+. However, it is very complicated because:

1. We need to do this upgrade in all the modules at once. If there are
Jersey 1.x and 2.x in the classpath, the application fails. This is
because we cannot upgrade Jersey module by module.
2. Jersey is used in many modules (HDFS, YARN, MapReduce)
3. There is no useful documentation for this upgrade. There is an
official migration guide, but it is poor.

I almost finished the work in Common and HDFS, but I'm facing some
errors in YARN unit tests (ex. TestNMWebServices). Jersey, Grizzly,
and Guice are used in the tests, and Guice is used for DI(Dependency
Injection). Jersey 2.x has its own DI module (HK2), so now there are
two DI modules. Although there is a bridge (HK2-bridge) module to use
both HK2 and Guice simultaneously, I cannot correctly set up the
environment in the unit tests.

Suppose you are good at Jersey 2.x + Guice, please help me. If there
is no help, I consider removing Guice and use the only HK2 for DI.

Thanks and regards,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC3

[Akira Ajisaka]

In addition, backported HDFS-13404 to fix
TestRouterWebHDFSContractAppend failure.

On Fri, Dec 4, 2020 at 10:43 AM Akira Ajisaka  wrote:
>
> I found TestCombineFileInputFormat is failing in branch-3.2.2 and I've
> cherry-picked MAPREDUCE-7284 to fix the failure.
>
> Thanks,
> Akira
>
> On Fri, Dec 4, 2020 at 9:09 AM Chao Sun  wrote:
> >
> > Thanks Xiaoqiao. I just committed the fix to branch-3.2 (and will port it
> > to other branches as well later). Could you cherry-pick it to branch-3.2.2
> > and start another RC? really appreciate it.
> >
> > Best,
> > Chao
> >
> > On Wed, Dec 2, 2020 at 7:59 PM Xiaoqiao He  wrote:
> >
> > > Thanks Chao Sun for your quick response.
> > >
> > > If no more comments, I will wait for HADOOP-16080 to be resolved and
> > > prepare another RC.
> > > BTW, If any other issues should be involved for 3.2.2, please let me know
> > > ASAP.  Thanks again.
> > >
> > > Regards.
> > > - He Xiaoqiao
> > >
> > > On Wed, Dec 2, 2020 at 2:55 PM Chao Sun  wrote:
> > >
> > >> Thanks Xiaoqiao for the work! Unfortunately we just discovered an issue
> > >> (HADOOP-16080) which prevents Spark from consuming this release. I'm
> > >> working on a PR for this and wondering if we can include the fix in the
> > >> release too.
> > >>
> > >> Thanks again and apologies for the late notice.
> > >>
> > >> Best,
> > >> Chao
> > >>
> > >> On Mon, Nov 30, 2020 at 7:37 AM Xiaoqiao He 
> > >> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> The release candidate (RC3) for Hadoop-3.2.2 is available now.
> > >>> There are 22 commits[1] differences between RC3 and RC2[2].
> > >>>
> > >>> The RC3 is available at:
> > >>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC3
> > >>> The RC3 tag in github is here:
> > >>> https://github.com/apache/hadoop/tree/release-3.2.2-RC3
> > >>> The maven artifacts are staged at:
> > >>> https://repository.apache.org/content/repositories/orgapachehadoop-1291
> > >>>
> > >>> You can find my public key at:
> > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> > >>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> > >>>
> > >>> Please try the release and vote.
> > >>>
> > >>> Thanks,
> > >>> He Xiaoqiao
> > >>>
> > >>> [1]
> > >>>
> > >>> https://github.com/apache/hadoop/compare/release-3.2.2-RC2...release-3.2.2-RC3
> > >>> [2]
> > >>>
> > >>> https://lists.apache.org/thread.html/r606fff445847bdb85bd60c5a73b2fb1f0433ee31b18c456a2231fcec%40%3Chdfs-dev.hadoop.apache.org%3E
> > >>> [3]
> > >>>
> > >>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
> > >>>
> > >>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC3

[Akira Ajisaka]

I found TestCombineFileInputFormat is failing in branch-3.2.2 and I've
cherry-picked MAPREDUCE-7284 to fix the failure.

Thanks,
Akira

On Fri, Dec 4, 2020 at 9:09 AM Chao Sun  wrote:
>
> Thanks Xiaoqiao. I just committed the fix to branch-3.2 (and will port it
> to other branches as well later). Could you cherry-pick it to branch-3.2.2
> and start another RC? really appreciate it.
>
> Best,
> Chao
>
> On Wed, Dec 2, 2020 at 7:59 PM Xiaoqiao He  wrote:
>
> > Thanks Chao Sun for your quick response.
> >
> > If no more comments, I will wait for HADOOP-16080 to be resolved and
> > prepare another RC.
> > BTW, If any other issues should be involved for 3.2.2, please let me know
> > ASAP.  Thanks again.
> >
> > Regards.
> > - He Xiaoqiao
> >
> > On Wed, Dec 2, 2020 at 2:55 PM Chao Sun  wrote:
> >
> >> Thanks Xiaoqiao for the work! Unfortunately we just discovered an issue
> >> (HADOOP-16080) which prevents Spark from consuming this release. I'm
> >> working on a PR for this and wondering if we can include the fix in the
> >> release too.
> >>
> >> Thanks again and apologies for the late notice.
> >>
> >> Best,
> >> Chao
> >>
> >> On Mon, Nov 30, 2020 at 7:37 AM Xiaoqiao He 
> >> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> The release candidate (RC3) for Hadoop-3.2.2 is available now.
> >>> There are 22 commits[1] differences between RC3 and RC2[2].
> >>>
> >>> The RC3 is available at:
> >>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC3
> >>> The RC3 tag in github is here:
> >>> https://github.com/apache/hadoop/tree/release-3.2.2-RC3
> >>> The maven artifacts are staged at:
> >>> https://repository.apache.org/content/repositories/orgapachehadoop-1291
> >>>
> >>> You can find my public key at:
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> >>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> >>>
> >>> Please try the release and vote.
> >>>
> >>> Thanks,
> >>> He Xiaoqiao
> >>>
> >>> [1]
> >>>
> >>> https://github.com/apache/hadoop/compare/release-3.2.2-RC2...release-3.2.2-RC3
> >>> [2]
> >>>
> >>> https://lists.apache.org/thread.html/r606fff445847bdb85bd60c5a73b2fb1f0433ee31b18c456a2231fcec%40%3Chdfs-dev.hadoop.apache.org%3E
> >>> [3]
> >>>
> >>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
> >>>
> >>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7305) [JDK 11] TestMRJobsWithProfiler fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7305:


 Summary: [JDK 11] TestMRJobsWithProfiler fails
 Key: MAPREDUCE-7305
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7305
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
 Environment: OpenJDK 11.0.9
Reporter: Akira Ajisaka


{noformat}
[INFO] Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
[ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 85.232 
s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
[ERROR] 
testDefaultProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)  
Time elapsed: 30.901 s  <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
at 
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDefaultProfiler(TestMRJobsWithProfiler.java:111)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)

[ERROR] 
testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)  
Time elapsed: 27.956 s  <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
at 
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC1

[Akira Ajisaka]

-1

- YARN resource localization is broken by HADOOP-17306 and it has been
reverted. It should be reverted from 3.2.2 as well. (Thank you Jim
Brennan for the report!)
- Would you include HDFS-15643 in RC2? This fixes checksum error in EC
with ISA-L.

Thank you He Xiaoqiao for preparing the release candidates.

-Akira

On Tue, Nov 3, 2020 at 7:48 PM Xiaoqiao He  wrote:
>
> Hi folks,
>
> I have put together a release candidate (RC1) for Hadoop-3.2.2.
>
> It contains *473[1]* fixed jira issues since 3.2.1 release which also
> include many features and improvements(Please reference the full set of
> release notes).
>
> The RC is available at:
> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC1
> The RC tag in github is here:
> https://github.com/apache/hadoop/tree/release-3.2.2-RC1
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1286
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS or
> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>
> Please try the release and vote. The vote will run for 5 days until
> 2020/11/09 at 00:00 CST.
>
> I have done a simple testing with my pseudo cluster.
> * Verify gpg sign and md5sum.
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple Shell/Job.
> * Check the web UI of
> NameNode/DFSRouter/DataNode/Resourcemanager/NodeManager.
> My +1 to start.
>
> NOTE:
> A. Explanation about RC1 rather than RC0. After code freeze for RC0 there
> are several new critical issues merged so create RC1.
> B. Thanks Sunil/Wei-Chiu/Sammi Chen/Eric Badger for your very helpful guide.
>
> Thanks
> He Xiaoqiao
>
> [1]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Fixing flaky tests in Apache Hadoop

[Akira Ajisaka]

Hi Hadoop developers,

Now there are a lot of failing unit tests and there is an issue to
tackle this bad situation.
https://issues.apache.org/jira/browse/HDFS-15646

Although this issue is in HDFS project, this issue is related to all
the Hadoop developers. Please check the above URL, read the
description, and volunteer to dedicate more time to fix flaky tests.
Your contribution to fixing the flaky tests will be really
appreciated!

Thank you Ahmed Hussein for your report.

Regards,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Precommit job fails to write comments in the GitHub pull requests

[Akira Ajisaka]

> Can we point to a commit before Yetus-994 instead of master and get away with 
> this?

I'm fine with that approach not to breaking more.
Filed https://issues.apache.org/jira/browse/HADOOP-17297

Thanks,
Akira

On Mon, Oct 5, 2020 at 10:18 AM Akira Ajisaka  wrote:
>
> Hi Mastake and Ayush,
>
> After YETUS-994, Yetus updates the commit status instead of writing a comment.
> Now I think Yetus still fail to update the commit status, however, now
> Jenkins can successfully update the commit status instead and the
> result is shown in the link.
> Example: 
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-2278/11/pipeline/
> Isn't it work for you?
>
> > I prefer using fixed/static version of toolchain in order to avoid 
> > surprising the developers.
> Agreed, but now there is no way to run javadoc successfully with Yetus
> 0.12.0 + Java 11.
> Ref: https://issues.apache.org/jira/browse/YETUS-972
>
> Now I'm trying to fix the error in Yetus.
>
> -Akira
>
> On Sat, Oct 3, 2020 at 3:25 AM Ayush Saxena  wrote:
> >
> > Hi Akira,
> > Thanx for working on this.
> > Is there anything blocking here? I think it isn’t working still.
> > Can we point to a commit before Yetus-994 instead of master and get away 
> > with this? or if there is no solution handy we may go ahead with approach 
> > #4 and then figure out how to proceed.
> >
> > -Ayush
> >
> > > On 29-Sep-2020, at 5:31 PM, Masatake Iwasaki 
> > >  wrote:
> > >
> > > Hi Akira,
> > >
> > > Thanks for working on this.
> > > It looks like you are trying token on PR #2348.
> > > https://github.com/apache/hadoop/pull/2348
> > >
> > > It it does not work, I think option #4 is reasonable.
> > > I prefer using fixed/static version of toolchain in order to
> > > avoid surprising the developers.
> > > We can upgrade Yetus when JDK11 Javadoc issue is fixed.
> > >
> > > Regards,
> > > Masatake Iwasaki
> > >
> > >> On 2020/09/29 2:55, Akira Ajisaka wrote:
> > >> Apache Hadoop developers,
> > >> After YETUS-994, the jenkins job updates the commit status instead of
> > >> writing a comment to the pull request. It requires an OAuth token with
> > >> write access to "repo:status" but now there is no such token for
> > >> Apache Hadoop. I asked the infra team to create a token:
> > >> https://issues.apache.org/jira/browse/INFRA-20906
> > >> For now, I think there 4 options to get the information:
> > >> 1. Attach a patch to the JIRA instead of GitHub
> > >> 2. Update the Jenkinsfile for a while (Sample PR:
> > >> https://github.com/apache/hadoop/pull/2346)
> > >> 3. Search the job and get the comment from
> > >> https://ci-hadoop.apache.org/job/hadoop-multibranch/
> > >> 4. Create a JIRA to update the Jenkinsfile to use 0.12.0 instead of
> > >> main. (JDK11 javadoc support is broken in Yetus 0.12.0)
> > >> Sorry for the inconvenience.
> > >> Regards,
> > >> Akira
> > >> -
> > >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Precommit job fails to write comments in the GitHub pull requests

[Akira Ajisaka]

Hi Mastake and Ayush,

After YETUS-994, Yetus updates the commit status instead of writing a comment.
Now I think Yetus still fail to update the commit status, however, now
Jenkins can successfully update the commit status instead and the
result is shown in the link.
Example: 
https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-2278/11/pipeline/
Isn't it work for you?

> I prefer using fixed/static version of toolchain in order to avoid surprising 
> the developers.
Agreed, but now there is no way to run javadoc successfully with Yetus
0.12.0 + Java 11.
Ref: https://issues.apache.org/jira/browse/YETUS-972

Now I'm trying to fix the error in Yetus.

-Akira

On Sat, Oct 3, 2020 at 3:25 AM Ayush Saxena  wrote:
>
> Hi Akira,
> Thanx for working on this.
> Is there anything blocking here? I think it isn’t working still.
> Can we point to a commit before Yetus-994 instead of master and get away with 
> this? or if there is no solution handy we may go ahead with approach #4 and 
> then figure out how to proceed.
>
> -Ayush
>
> > On 29-Sep-2020, at 5:31 PM, Masatake Iwasaki  
> > wrote:
> >
> > Hi Akira,
> >
> > Thanks for working on this.
> > It looks like you are trying token on PR #2348.
> > https://github.com/apache/hadoop/pull/2348
> >
> > It it does not work, I think option #4 is reasonable.
> > I prefer using fixed/static version of toolchain in order to
> > avoid surprising the developers.
> > We can upgrade Yetus when JDK11 Javadoc issue is fixed.
> >
> > Regards,
> > Masatake Iwasaki
> >
> >> On 2020/09/29 2:55, Akira Ajisaka wrote:
> >> Apache Hadoop developers,
> >> After YETUS-994, the jenkins job updates the commit status instead of
> >> writing a comment to the pull request. It requires an OAuth token with
> >> write access to "repo:status" but now there is no such token for
> >> Apache Hadoop. I asked the infra team to create a token:
> >> https://issues.apache.org/jira/browse/INFRA-20906
> >> For now, I think there 4 options to get the information:
> >> 1. Attach a patch to the JIRA instead of GitHub
> >> 2. Update the Jenkinsfile for a while (Sample PR:
> >> https://github.com/apache/hadoop/pull/2346)
> >> 3. Search the job and get the comment from
> >> https://ci-hadoop.apache.org/job/hadoop-multibranch/
> >> 4. Create a JIRA to update the Jenkinsfile to use 0.12.0 instead of
> >> main. (JDK11 javadoc support is broken in Yetus 0.12.0)
> >> Sorry for the inconvenience.
> >> Regards,
> >> Akira
> >> -
> >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7289) Fix wrong comment in LongLong.java

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7289.
--
Fix Version/s: 2.10.2
   3.3.1
   3.1.5
   3.4.0
   3.2.2
 Hadoop Flags: Reviewed
 Assignee: Wanqiang Ji  (was: Siddharth Ahuja)
   Resolution: Fixed

Committed to all the active branches. Thank you [~jiwq].

> Fix wrong comment in LongLong.java
> --
>
> Key: MAPREDUCE-7289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7289
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation, examples
>    Reporter: Akira Ajisaka
>Assignee: Wanqiang Ji
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1, 2.10.2
>
>
> {code:title=LongLong.java}
>   /** Shift right operation (<<). */
> {code}
> Right shift is {{>>}}.
> Found when reviewing MAPREDUCE-7288



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Precommit job fails to write comments in the GitHub pull requests

[Akira Ajisaka]

I've got a GitHub App token from the infra team and set this to the
hadoop-multibranch job.
The commit statuses in GitHub pull requests will be updated by Jenkins
and you'll need to check these statuses.

-Akira

On Tue, Sep 29, 2020 at 2:55 AM Akira Ajisaka  wrote:
>
> Apache Hadoop developers,
>
> After YETUS-994, the jenkins job updates the commit status instead of
> writing a comment to the pull request. It requires an OAuth token with
> write access to "repo:status" but now there is no such token for
> Apache Hadoop. I asked the infra team to create a token:
> https://issues.apache.org/jira/browse/INFRA-20906
>
> For now, I think there 4 options to get the information:
>
> 1. Attach a patch to the JIRA instead of GitHub
> 2. Update the Jenkinsfile for a while (Sample PR:
> https://github.com/apache/hadoop/pull/2346)
> 3. Search the job and get the comment from
> https://ci-hadoop.apache.org/job/hadoop-multibranch/
> 4. Create a JIRA to update the Jenkinsfile to use 0.12.0 instead of
> main. (JDK11 javadoc support is broken in Yetus 0.12.0)
>
> Sorry for the inconvenience.
>
> Regards,
> Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Precommit job fails to write comments in the GitHub pull requests

[Akira Ajisaka]

Apache Hadoop developers,

After YETUS-994, the jenkins job updates the commit status instead of
writing a comment to the pull request. It requires an OAuth token with
write access to "repo:status" but now there is no such token for
Apache Hadoop. I asked the infra team to create a token:
https://issues.apache.org/jira/browse/INFRA-20906

For now, I think there 4 options to get the information:

1. Attach a patch to the JIRA instead of GitHub
2. Update the Jenkinsfile for a while (Sample PR:
https://github.com/apache/hadoop/pull/2346)
3. Search the job and get the comment from
https://ci-hadoop.apache.org/job/hadoop-multibranch/
4. Create a JIRA to update the Jenkinsfile to use 0.12.0 instead of
main. (JDK11 javadoc support is broken in Yetus 0.12.0)

Sorry for the inconvenience.

Regards,
Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Moving Ozone to a separated Apache project

[Akira Ajisaka]

+1

Thanks,
Akira

On Fri, Sep 25, 2020 at 3:00 PM Elek, Marton  wrote:
>
> Hi all,
>
> Thank you for all the feedback and requests,
>
> As we discussed in the previous thread(s) [1], Ozone is proposed to be a
> separated Apache Top Level Project (TLP)
>
> The proposal with all the details, motivation and history is here:
>
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
>
> This voting runs for 7 days and will be concluded at 2nd of October, 6AM
> GMT.
>
> Thanks,
> Marton Elek
>
> [1]:
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] fate of branch-2.9

[Akira Ajisaka]

+1

Thanks,
Akira

On Fri, Aug 28, 2020 at 5:51 PM lisheng.sun08 
wrote:

> +1 on EOL of branch-2.9.
>
> Thanks,
> Lisheng Sun
>
>
>
> 发自我的小米手机
> 在 2020年8月27日 下午1:55，Wei-Chiu Chuang 写道：
>
> Bump up this thread after 6 months.
>
> Is anyone still interested in the 2.9 release line? Or are we good to
> start
> the EOL process? The 2.9.2 was released in Nov 2018.
>
> I'd really like to see the community to converge to fewer release lines
> and
> make more frequent releases in each line.
>
> Thanks,
> Weichiu
>
>
> On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang 
> wrote:
>
> > I think that's a great suggestion.
> > Currently, we make 1 minor release per year, and within each minor
> release
> > we bring up 1 thousand to 2 thousand commits in it compared with the
> > previous one.
> > I can totally understand it is a big bite for users to swallow. Having a
> > more frequent release cycle, plus LTS and non-LTS releases should help
> with
> > this. (Of course we will need to make the release preparation much
> easier,
> > which is currently a pain)
> >
> > I am happy to discuss the release model further in the dev ML. LTS v.s.
> > non-LTS is one suggestion.
> >
> > Another similar issue: In the past Hadoop strived to
> > maintain compatibility. However, this is no longer sustainable as more
> CVEs
> > coming from our dependencies: netty, jetty, jackson ... etc.
> > In many cases, updating the dependencies brings breaking changes. More
> > recently, especially in Hadoop 3.x, I started to make the effort to
> update
> > dependencies much more frequently. How do users feel about this change?
> >
> > On Thu, Mar 5, 2020 at 7:58 AM Igor Dvorzhak 
> > wrote:
> >
> >> Maybe Hadoop will benefit from adopting a similar release and support
> >> strategy as Java? I.e. designate some releases as LTS and support them
> for
> >> 2 (?) years (it seems that 2.7.x branch was de-facto LTS), other
> non-LTS
> >> releases will be supported for 6 months (or until next release). This
> >> should allow to reduce maintenance cost of non-LTS release and provide
> >> conservative users desired stability by allowing them to wait for new
> LTS
> >> release and upgrading to it.
> >>
> >> On Thu, Mar 5, 2020 at 1:26 AM Rupert Mazzucco <
> rupert.mazzu...@gmail.com>
> >> wrote:
> >>
> >>> After recently jumping from 2.7.7 to 2.10 without issue myself, I vote
> >>> for keeping only the 2.10 line.
> >>> It would seem all other 2.x branches can upgrade to a 2.10.x easily if
> >>> they feel like upgrading at all,
> >>> unlike a jump to 3.x, which may require more planning.
> >>>
> >>> I also vote for having only one main 3.x branch. Why are there 3.1.x
> and
> >>> 3.2.x seemingly competing,
> >>> and now 3.3.x? For a community that does not have the resources to
> >>> manage multiple release lines,
> >>> you guys sure like to multiply release lines a lot.
> >>>
> >>> Cheers
> >>> Rupert
> >>>
> >>> Am Mi., 4. März 2020 um 19:40 Uhr schrieb Wei-Chiu Chuang
> >>> :
> >>>
>  Forwarding the discussion thread from the dev mailing lists to the
> user
>  mailing lists.
> 
>  I'd like to get an idea of how many users are still on Hadoop 2.9.
>  Please share your thoughts.
> 
>  On Mon, Mar 2, 2020 at 6:30 PM Sree Vaddi
>   wrote:
> 
> > +1
> >
> > Sent from Yahoo Mail on Android
> >
> >   On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang
>
> > wrote:   Hi,
> >
> > Following the discussion to end branch-2.8, I want to start a
> > discussion
> > around what's next with branch-2.9. I am hesitant to use the word
> "end
> > of
> > life" but consider these facts:
> >
> > * 2.9.0 was released Dec 17, 2017.
> > * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is more
> > than
> > 15 months ago.
> > * no one seems to be interested in being the release manager for
> 2.9.3.
> > * Most if not all of the active Hadoop contributors are using Hadoop
> > 2.10
> > or Hadoop 3.x.
> > * We as a community do not have the cycle to manage multiple release
> > line,
> > especially since Hadoop 3.3.0 is coming out soon.
> >
> > It is perhaps the time to gradually reduce our footprint in Hadoop
> > 2.x, and
> > encourage people to upgrade to Hadoop 3.x
> >
> > Thoughts?
> >
> >
>
> - To
> unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional
> commands, e-mail: user-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7289) Fix wrong comment in LongLong.java

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7289:


 Summary: Fix wrong comment in LongLong.java
 Key: MAPREDUCE-7289
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7289
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: examples
Reporter: Akira Ajisaka


{code:title=LongLong.java}
  /** Shift right operation (<<). */
{code}
Right shift is {{>>}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-7051) Fix typo in MultipleOutputFormat

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7051.
--
Fix Version/s: 3.3.1
   3.1.5
   3.4.0
   3.2.2
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to trunk, branch-3.3, branch-3.2, and branch-3.1. Thanks [~ywheel] 
for your contribution!

> Fix typo in MultipleOutputFormat
> 
>
> Key: MAPREDUCE-7051
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7051
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: ywheel
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.2.2, 3.4.0, 3.1.5, 3.3.1
>
> Attachments: MAPREDUCE-7051.patch
>
>
> In org.apache.hadoop.mapred.lib.MultipleOutputFormat, there is a typo for the 
> java doc of getInputFileBasedOutputFileName method. 
> "the outfile name based on a given anme and the input file name" should be 
> "the outfile name based on a given name and the input file name"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7284) TestCombineFileInputFormat#testMissingBlocks fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7284:


 Summary: TestCombineFileInputFormat#testMissingBlocks fails
 Key: MAPREDUCE-7284
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7284
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Akira Ajisaka


o.a.h.mapreduce.lib.input.TestCombineFileInputFormat#testMissingBlocks fails on 
trunk:
{noformat}
[INFO] Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
[ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.916 
s <<< FAILURE! - in 
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
[ERROR] 
testMissingBlocks(org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat)
  Time elapsed: 0.59 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem 
cannot be cast to 
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat$MissingBlockFileSystem
at 
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat.testMissingBlocks(TestCombineFileInputFormat.java:1655)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0

[Akira Ajisaka]

+1 (binding)

- Verified checksums and signatures.
- Built from the source with CentOS 7 and OpenJDK 8.
- Successfully upgraded HDFS to 3.3.0-RC0 in our development cluster (with
RBF, security, and OpenJDK 11) for end-users. No issues reported.
- The document looks good.
- Deployed pseudo cluster and ran some MapReduce jobs.

Thanks,
Akira


On Tue, Jul 7, 2020 at 7:27 AM Brahma Reddy Battula 
wrote:

> Hi folks,
>
> This is the first release candidate for the first release of Apache
> Hadoop 3.3.0
> line.
>
> It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of
> features and improvements(read the full set of release notes).
>
> Below feature additions are the highlights of the release.
>
> - ARM Support
> - Enhancements and new features on S3a,S3Guard,ABFS
> - Java 11 Runtime support and TLS 1.3.
> - Support Tencent Cloud COS File System.
> - Added security to HDFS Router.
> - Support non-volatile storage class memory(SCM) in HDFS cache directives
> - Support Interactive Docker Shell for running Containers.
> - Scheduling of opportunistic containers
> - A pluggable device plugin framework to ease vendor plugin development
>
> *The RC0 artifacts are at*:
> http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/
>
> *First release to include ARM binary, Have a check.*
> *RC tag is *release-3.3.0-RC0.
>
>
> *The maven artifacts are hosted here:*
> https://repository.apache.org/content/repositories/orgapachehadoop-1271/
>
> *My public key is available here:*
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST.
>
>
> I have done a few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
> Brahma Reddy Battula
>
>
> 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND
> fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY
> fixVersion ASC
>

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC1)

[Akira Ajisaka]

Hi Gabor,

Thank you for your work!

Kihwal reported IBR leak in standby NameNode:
https://issues.apache.org/jira/browse/HDFS-15421.
I think this is a blocker and this affects 3.1.4-RC1. Would you check this?

Best regards,
Akira

On Mon, Jun 22, 2020 at 10:26 PM Gabor Bota 
wrote:

> Hi folks,
>
> I have put together a release candidate (RC1) for Hadoop 3.1.4.
>
> The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC1/
> The RC tag in git is here:
> https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC1
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1267/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C
>
> Please try the release and vote. The vote will run for 5 weekdays,
> until June 30. 2020. 23:00 CET.
>
> Thanks,
> Gabor
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Moving to Yetus 0.12.0

[Akira Ajisaka]

Updated Yetus version to 0.12.0 in branch-2.10 qbt job:
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch-2.10-java7-linux-x86/

-Akira

On Tue, Apr 28, 2020 at 6:59 AM Akira Ajisaka  wrote:

> Hi folks,
>
> After upgrading the image from Ubuntu 16.04 to Ubuntu 18.04 (
> https://issues.apache.org/jira/browse/HADOOP-16054), there are 1000+
> failed unit tests in the daily qbt job (
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1482/console).
> On the other hand, my testing qbt job in the new CI servers (with Yetus
> 0.12.0) is still working well, so I configured the new qbt job (
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/) to
> send e-mails to the dev lists. I'll disable the old qbt job in a few days
> if there are no problems.
>
> Regards,
> Akira
>
> On Sun, Apr 19, 2020 at 5:05 PM Akira Ajisaka  wrote:
>
>> Updated Precommit-(HADOOP|HDFS|YARN|MAPREDUCE)-Build jobs. The daily qbt
>> jobs are kept as-is. Now I'm testing the new CI servers and testing the qbt
>> jobs with Yetus 0.12.0.
>> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86 (actually
>> the test runs on Java 8, I'll rename it)
>>
>> -Akira
>>
>> On Sun, Apr 19, 2020 at 3:41 AM Akira Ajisaka 
>> wrote:
>>
>>> Hi folks,
>>>
>>> I updated Jenkinsfile to use Yetus 0.12.0 in GitHub PR.
>>> https://issues.apache.org/jira/browse/HADOOP-16944
>>>
>>> In addition, I'm updating the configs in the Jenkins jobs to use Yetus
>>> 0.12.0 in ASF JIRAs. I updated the settings of
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build
>>> <https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/16898/console>
>>>  and
>>> testing in https://issues.apache.org/jira/browse/HADOOP-17000.
>>>
>>> If there is something wrong, please let me know and please feel free to
>>> revert the config.
>>>
>>> Regards,
>>> Akira
>>>
>>

Re: [DISCUSS] making Ozone a separate Apache project

[Akira Ajisaka]

+1

-Akira

On Wed, May 13, 2020 at 4:53 PM Elek, Marton  wrote:

>
>
> I would like to start a discussion to make a separate Apache project for
> Ozone
>
>
>
> ### HISTORY [1]
>
>   * Apache Hadoop Ozone development started on a feature branch of
> Hadoop repository (HDFS-7240)
>
>   * In the October of 2017 a discussion has been started to merge it to
> the Hadoop main branch
>
>   * After a long discussion it's merged to Hadoop trunk at the March of
> 2018
>
>   * During the discussion of the merge, it was suggested multiple times
> to create a separated project for the Ozone. But at that time:
>  1). Ozone was tightly integrated with Hadoop/HDFS
>  2). There was an active plan to use Block layer of Ozone (HDDS or
> HDSL at that time) as the block level of HDFS
>  3). The community of Ozone was a subset of the HDFS community
>
>   * The first beta release of Ozone was just released. Seems to be a
> good time before the first GA to make a decision about the future.
>
>
>
> ### WHAT HAS BEEN CHANGED
>
>   During the last years Ozone became more and more independent both at
> the community and code side. The separation has been suggested again and
> again (for example by Owen [2] and Vinod [3])
>
>
>
>   From COMMUNITY point of view:
>
>
>* Fortunately more and more new contributors are helping Ozone.
> Originally the Ozone community was a subset of HDFS project. But now a
> bigger and bigger part of the community is related to Ozone only.
>
>* It seems to be easier to _build_ the community as a separated project.
>
>* A new, younger project might have different practices
> (communication, commiter criteria, development style) compared to old,
> mature project
>
>* It's easier to communicate (and improve) these standards in a
> separated projects with clean boundaries
>
>* Separated project/brand can help to increase the adoption rate and
> attract more individual contributor (AFAIK it has been seen in Submarine
> after a similar move)
>
>   * Contribution process can be communicated more easily, we can make
> first time contribution more easy
>
>
>
>   From CODE point of view Ozone became more and more independent:
>
>
>   * Ozone has different release cycle
>
>   * Code is already separated from Hadoop code base
> (apache/hadoop-ozone.git)
>
>   * It has separated CI (github actions)
>
>   * Ozone uses different (more strict) coding style (zero toleration of
> unit test / checkstyle errors)
>
>   * The code itself became more and more independent from Hadoop on
> Maven level. Originally it was compiled together with the in-tree latest
> Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> Configuration...)
>
>   * It starts to use multiple version of Hadoop (on client side)
>
>   * Volume of resolved issues are already very high on Ozone side (Ozone
> had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> together in the last 2-3 months)
>
>
> Summary: Before the first Ozone GA release, It seems to be a good time
> to discuss the long-term future of Ozone. Managing it as a separated TLP
> project seems to have more benefits.
>
>
> Please let me know what your opinion is...
>
> Thanks a lot,
> Marton
>
>
>
>
>
> [1]: For more details, see:
> https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
>
> [2]:
>
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
>
> [3]:
>
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Access to Hadoop QA account for migrating Precommit jobs in Hadoop

[Akira Ajisaka]

Hi Allen,

I'm testing new Jenkins master and find that we have to set hadoopqa JIRA
password to migrate the PreCommit-Admin and
PreCommit-(HADOOP|HDFS|YARN|MAPREDUCE)-Build. Would you migrate the jobs or
register the credential to the new Jenkins master (
https://ci-hadoop.apache.org/)?

Best regards,
Akira

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Akira Ajisaka]

Hi Surendra,

Updated the version to 3.3.1-SNAPSHOT in branch-3.3.

Thanks,
Akira

On Wed, Apr 29, 2020 at 4:22 PM Surendra Singh Lilhore <
surendralilh...@gmail.com> wrote:

> Hi Brahma,
>
> Why the branch-3.3 & branch-3.3.0 pom version is same ?.
>
> In branch-3.3 pom version should be 3.3.1.
>
> Please correct me if I am wrong.
>
> -Surendra
>
>
> On Sat, 25 Apr, 2020, 9:33 am Mingliang Liu,  wrote:
>
> > Brahma,
> >
> > What about https://issues.apache.org/jira/browse/HADOOP-17007?
> >
> > Thanks,
> >
> > On Fri, Apr 24, 2020 at 11:07 AM Brahma Reddy Battula  >
> > wrote:
> >
> > > Ok. Done. Branch created.
> > >
> > > Following blockers are pending, will closely track this.
> > >
> > > https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under
> > discussion
> > > )
> > > https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available)
> > > https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available)
> > > https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available)
> > >
> > >
> > > On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang
> > >  wrote:
> > >
> > > > +1 we should have the branch ASAP.
> > > >
> > > > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka 
> > > > wrote:
> > > >
> > > > > > Since blockers are not closed, I didn't cut the branch because
> > > > > multiple branches might confuse or sombody might miss to commit.
> > > > >
> > > > > The current situation is already confusing. The 3.3.1 version
> already
> > > > > exists in JIRA, so some committers wrongly commit non-critical
> issues
> > > to
> > > > > branch-3.3 and set the fix version to 3.3.1.
> > > > > I think now we should cut branch-3.3.0 and freeze source code
> except
> > > the
> > > > > blockers.
> > > > >
> > > > > -Akira
> > > > >
> > > > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula <
> > > bra...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Sure, I will do that.
> > > > >>
> > > > >> Since blockers are not closed, I didn't cut the branch because
> > > > >> multiple branches might confuse or sombody might miss to
> > commit.Shall
> > > I
> > > > >> wait till this weekend to create..?
> > > > >>
> > > > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka <
> aajis...@apache.org
> > >
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Brahma,
> > > > >>>
> > > > >>> Thank you for preparing the release.
> > > > >>> Could you cut branch-3.3.0? I would like to backport some fixes
> for
> > > > >>> 3.3.1 and not for 3.3.0.
> > > > >>>
> > > > >>> Thanks and regards,
> > > > >>> Akira
> > > > >>>
> > > > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula <
> > > > bra...@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi All,
> > > > >>>>
> > > > >>>> we are down to two blockers issues now (YARN-10194 and
> YARN-9848)
> > > > which
> > > > >>>> are in patch available state.Hopefully we can out the RC soon.
> > > > >>>>
> > > > >>>> thanks to @Prabhu Joseph 
> > > > ,@masakate,@akira
> > > > >>>> and @Wei-Chiu Chuang   and others for
> > helping
> > > > >>>> resloving the blockers.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula <
> > > > >>>> bra...@apache.org> wrote:
> > > > >>>>
> > > > >>>>>
> > > > >>>>> @Prabhu Joseph 
> > > > >>>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
> > > > >>>>> cherry-picked to branch-3.3. Right now, there are two blocker
> > > Jiras -
> > > > >>>

Re: Moving to Yetus 0.12.0

[Akira Ajisaka]

Hi folks,

After upgrading the image from Ubuntu 16.04 to Ubuntu 18.04 (
https://issues.apache.org/jira/browse/HADOOP-16054), there are 1000+ failed
unit tests in the daily qbt job (
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1482/console).
On the other hand, my testing qbt job in the new CI servers (with Yetus
0.12.0) is still working well, so I configured the new qbt job (
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/) to
send e-mails to the dev lists. I'll disable the old qbt job in a few days
if there are no problems.

Regards,
Akira

On Sun, Apr 19, 2020 at 5:05 PM Akira Ajisaka  wrote:

> Updated Precommit-(HADOOP|HDFS|YARN|MAPREDUCE)-Build jobs. The daily qbt
> jobs are kept as-is. Now I'm testing the new CI servers and testing the qbt
> jobs with Yetus 0.12.0.
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86 (actually
> the test runs on Java 8, I'll rename it)
>
> -Akira
>
> On Sun, Apr 19, 2020 at 3:41 AM Akira Ajisaka  wrote:
>
>> Hi folks,
>>
>> I updated Jenkinsfile to use Yetus 0.12.0 in GitHub PR.
>> https://issues.apache.org/jira/browse/HADOOP-16944
>>
>> In addition, I'm updating the configs in the Jenkins jobs to use Yetus
>> 0.12.0 in ASF JIRAs. I updated the settings of
>> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build
>> <https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/16898/console>
>>  and
>> testing in https://issues.apache.org/jira/browse/HADOOP-17000.
>>
>> If there is something wrong, please let me know and please feel free to
>> revert the config.
>>
>> Regards,
>> Akira
>>
>

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Akira Ajisaka]

> Since blockers are not closed, I didn't cut the branch because
multiple branches might confuse or sombody might miss to commit.

The current situation is already confusing. The 3.3.1 version already
exists in JIRA, so some committers wrongly commit non-critical issues to
branch-3.3 and set the fix version to 3.3.1.
I think now we should cut branch-3.3.0 and freeze source code except the
blockers.

-Akira

On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula 
wrote:

> Sure, I will do that.
>
> Since blockers are not closed, I didn't cut the branch because
> multiple branches might confuse or sombody might miss to commit.Shall I
> wait till this weekend to create..?
>
> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka 
> wrote:
>
>> Hi Brahma,
>>
>> Thank you for preparing the release.
>> Could you cut branch-3.3.0? I would like to backport some fixes for 3.3.1
>> and not for 3.3.0.
>>
>> Thanks and regards,
>> Akira
>>
>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula 
>> wrote:
>>
>>> Hi All,
>>>
>>> we are down to two blockers issues now (YARN-10194 and YARN-9848) which
>>> are in patch available state.Hopefully we can out the RC soon.
>>>
>>> thanks to @Prabhu Joseph  ,@masakate,@akira
>>> and @Wei-Chiu Chuang   and others for helping
>>> resloving the blockers.
>>>
>>>
>>>
>>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula 
>>> wrote:
>>>
>>>>
>>>> @Prabhu Joseph 
>>>> >>> Have committed the YARN blocker YARN-10219 to trunk and
>>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
>>>> YARN-10233 and HADOOP-16982
>>>> which i will help to review and commit. Thanks.
>>>>
>>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
>>>> blockers. (without YARN-10233 we have given so many releases,it's not newly
>>>> introduced.).. Thanks
>>>>
>>>> @Vinod Kumar Vavilapalli  ,@adam Antal,
>>>>
>>>> I noted YARN-9848 as a blocker as you mentioned above.
>>>>
>>>> @All,
>>>>
>>>> Currently following four blockers are pending for 3.3.0 RC.
>>>>
>>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
>>>> vino...@apache.org> wrote:
>>>>
>>>>> Looks like a really bad bug to me.
>>>>>
>>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should
>>>>> also revert it in a 3.2 maintenance release too.
>>>>>
>>>>> Thanks
>>>>> +Vinod
>>>>>
>>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal 
>>>>> > 
>>>>> wrote:
>>>>> >
>>>>> > Hi everyone,
>>>>> >
>>>>> > Sorry for coming a bit late with this, but there's also one jira
>>>>> that can
>>>>> > have potential impact on clusters and we should talk about it.
>>>>> >
>>>>> > Steven Rand found this problem earlier and commented to
>>>>> > https://issues.apache.org/jira/browse/YARN-4946.
>>>>> > The bug has impact on the RM state store: the RM does not delete
>>>>> apps - see
>>>>> > more details in his comment here:
>>>>> >
>>>>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>>>>> > .
>>>>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848
>>>>> with
>>>>> > the revert task).
>>>>> >
>>>>> > It might not be an actual blocker, but since there wasn't any
>>>>> consensus
>>>>> > about a follow up action, I thought we should decide how to proceed
>>>>> before
>>>>> > release 3.3.0.
>>>>> >
>>>>> > Regards,
>>>>> > Adam
>>>>> >
>>>>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>>>>> prabhujose.ga...@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> Thanks Brahma for the update.
>>>>> >&

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Akira Ajisaka]

t;>> <
>>> >>>
>>> https://issues.apache.org/jira/rest/api/1.0/issues/13231926/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
>>> >>>>
>>> >>>
>>> >>>
>>> >>> 1–5 of 5Refresh results
>>> >>> <
>>> >>>
>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20YARN%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
>>> >>>>
>>> >>> Columns
>>> >>> Patch InfoKeyTSummaryAssigneeReporterP
>>> StatusResolutionUpdatedDueCreated
>>> >>> <https://issues.apache.org/jira/browse/YARN-10001> YARN-10001
>>> >>> <https://issues.apache.org/jira/browse/YARN-10001> [image:
>>> Improvement]
>>> >>> <https://issues.apache.org/jira/browse/YARN-10001>
>>> >>>
>>> >>> Add explanation of unimplemented methods in
>>> InMemoryConfigurationStore
>>> >>> <https://issues.apache.org/jira/browse/YARN-10001>
>>> >>> Siddharth Ahuja
>>> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sahuja>
>>> >>> Szilard
>>> >>> Nemeth <
>>> >>> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
>>> >>> [image:
>>> >>> Major] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
>>> >>> <
>>> >>>
>>> https://issues.apache.org/jira/rest/api/1.0/issues/13271213/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
>>> >>>>
>>> >>> <https://issues.apache.org/jira/browse/YARN-9997> YARN-9997
>>> >>> <https://issues.apache.org/jira/browse/YARN-9997> [image:
>>> Improvement]
>>> >>> <https://issues.apache.org/jira/browse/YARN-9997>
>>> >>>
>>> >>> Code cleanup in ZKConfigurationStore
>>> >>> <https://issues.apache.org/jira/browse/YARN-9997>
>>> >>> Andras Gyori
>>> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gandras
>>> >
>>> >>> Szilard
>>> >>> Nemeth <
>>> >>> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
>>> >>> [image:
>>> >>> Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
>>> >>> <
>>> >>>
>>> https://issues.apache.org/jira/rest/api/1.0/issues/13271201/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
>>> >>>>
>>> >>> <https://issues.apache.org/jira/browse/YARN-10002> YARN-10002
>>> >>> <https://issues.apache.org/jira/browse/YARN-10002> [image:
>>> Improvement]
>>> >>> <https://issues.apache.org/jira/browse/YARN-10002>
>>> >>>
>>> >>> Code cleanup and improvements in ConfigurationStoreBaseTest
>>> >>> <https://issues.apache.org/jira/browse/YARN-10002>
>>> >>> Benjamin Teke
>>> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke>
>>> >>> Szilard
>>> >>> Nemeth <
>>> >>> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
>>> >>> [image:
>>> >>> Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
>>> >>> <
>>> >>>
>>> https://issues.apache.org/jira/rest/api/1.0/issues/13271215/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
>>> >>>>
>>> >>> <https://issues.apache.org/jira/browse/YARN-9998> YARN-9998
>>> >>> <https://issues.apache.org/jira/browse/YARN-9998> [image:
>>> Improvement]
>>> >>> <https://issues.apache.org/jira/browse/YARN-9998>
>>> >>>
>>> >>> Code cleanup in LeveldbConfigurationStore
>>> >>> <https://issues.apache.org/jira/browse/YARN-9998>
>>> >>> Benjamin Teke
>>> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke>
>>> >>> Szilard
>>> >>> Nemeth <
>>> >>> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth>
>>> >>> [image:
>>> >>> Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20   28/Nov/19 Actions
>>> >>> <
>>> >>>
>>> https://issues.apache.org/jira/rest/api/1.0/issues/13271202/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin
>>> >>>>
>>> >>> <https://issues.apache.org/jira/browse/YARN-9354> YARN-9354
>>> >>> <https://issues.apache.org/jira/browse/YARN-9354> [image:
>>> Improvement]
>>> >>> <https://issues.apache.org/jira/browse/YARN-9354>
>>> >>>
>>> >>> Resources sh <https://issues.apache.org/jira/browse/YARN-9354>
>>> >>>
>>> >>> On Thu, Apr 9, 2020 at 11:41 PM Brahma Reddy Battula <
>>> bra...@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> thanks a lot Akira and Prabhu for ping.
>>> >>>>
>>> >>>> All the blockers/critical issues are fixed. Recently two are
>>> reported
>>> >>>> again like HADOOP-16955 and HADOOP-16963 waiting for confirmation.
>>> >>>>
>>> >>>> @prabhu, Those 156 jira's are non-blocker issues which needs to
>>> moved to
>>> >>>> 3.4.0. and filter for blocker/critical is following.
>>> >>>>
>>> >>>> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
>>> >>>> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0
>>> >>> ORDER
>>> >>>> BY priority DESC
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph <
>>> >>> prabhujose.ga...@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi Brahma,
>>> >>>>>
>>> >>>>> Please let me know if i can be of any help in review and
>>> commit /
>>> >>>>> clear the pending issues in YARN and MapReduce targeted for 3.3.0.
>>> >>>>> I see there are 156 YARN + MapReduce Unresolved Jiras which are
>>> either
>>> >>>>> with
>>> >>>>> target or fix version set to 3.3.0.
>>> >>>>>
>>> >>>>> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND
>>> >>> (cf[12310320]
>>> >>>>> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Prabhu Joseph
>>> >>>>>
>>> >>>>> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka 
>>> >>> wrote:
>>> >>>>>
>>> >>>>>> Hi Brahma,
>>> >>>>>>
>>> >>>>>> How is this issue going?
>>> >>>>>> If there are some blockers, I can help you.
>>> >>>>>>
>>> >>>>>> -Akira
>>> >>>>>>
>>> >>>>>> On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula <
>>> >>> bra...@apache.org
>>> >>>>>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hi All,
>>> >>>>>>>
>>> >>>>>>> branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
>>> >>>>>>>
>>> >>>>>>> while committing jira's please set the proper fix version and if
>>> >>>>> there is
>>> >>>>>>> any blocker/critical please commit to branch-3.3 or let me know.
>>> >>>>>>>
>>> >>>>>>> Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
>>> >>>>>>> communication in another thread[2]).
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> 1.https://github.com/apache/hadoop/commits/branch-3.3
>>> >>>>>>> 2.
>>> >>>>>
>>> https://lists.apache.org/list.html?common-...@hadoop.apache.org:2020-3
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> --Brahma Reddy Battula
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --Brahma Reddy Battula
>>> >>>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>>
>>> >>>
>>> >>>
>>> >>> --Brahma Reddy Battula
>>> >>>
>>> >>
>>>
>>>
>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: Moving to Yetus 0.12.0

[Akira Ajisaka]

Updated Precommit-(HADOOP|HDFS|YARN|MAPREDUCE)-Build jobs. The daily qbt
jobs are kept as-is. Now I'm testing the new CI servers and testing the qbt
jobs with Yetus 0.12.0.
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86 (actually
the test runs on Java 8, I'll rename it)

-Akira

On Sun, Apr 19, 2020 at 3:41 AM Akira Ajisaka  wrote:

> Hi folks,
>
> I updated Jenkinsfile to use Yetus 0.12.0 in GitHub PR.
> https://issues.apache.org/jira/browse/HADOOP-16944
>
> In addition, I'm updating the configs in the Jenkins jobs to use Yetus
> 0.12.0 in ASF JIRAs. I updated the settings of
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build
> <https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/16898/console>
>  and
> testing in https://issues.apache.org/jira/browse/HADOOP-17000.
>
> If there is something wrong, please let me know and please feel free to
> revert the config.
>
> Regards,
> Akira
>

Moving to Yetus 0.12.0

[Akira Ajisaka]

Hi folks,

I updated Jenkinsfile to use Yetus 0.12.0 in GitHub PR.
https://issues.apache.org/jira/browse/HADOOP-16944

In addition, I'm updating the configs in the Jenkins jobs to use Yetus
0.12.0 in ASF JIRAs. I updated the settings of
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build

and
testing in https://issues.apache.org/jira/browse/HADOOP-17000.

If there is something wrong, please let me know and please feel free to
revert the config.

Regards,
Akira

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Akira Ajisaka]

Hi folks,

I am still seeing some changes are being committed to branch-2.
I'd like to delete the source code from branch-2 to avoid mistakes.
https://issues.apache.org/jira/browse/HADOOP-16988

-Akira

On Wed, Jan 1, 2020 at 2:38 AM Ayush Saxena  wrote:

> Hi Jim,
> Thanx for catching, I have configured the build to run on branch-2.10.
>
> -Ayush
>
> On Tue, 31 Dec 2019 at 22:50, Jim Brennan 
> wrote:
>
>> It looks like QBT tests are still being run on branch-2 (
>> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/),
>> and they are not very helpful at this point.
>> Can we change the QBT tests to run against branch-2.10 instead?
>>
>> Jim
>>
>> On Mon, Dec 23, 2019 at 7:44 PM Akira Ajisaka 
>> wrote:
>>
>>> Thank you, Ayush.
>>>
>>> I understand we should keep branch-2 as is, as well as master.
>>>
>>> -Akira
>>>
>>>
>>> On Mon, Dec 23, 2019 at 9:14 PM Ayush Saxena  wrote:
>>>
>>> > Hi Akira
>>> > Seems there was an INFRA ticket for that. INFRA-19581,
>>> > But the INFRA people closed as wont do and yes, the branch is
>>> protected,
>>> > we can’t delete it directly.
>>> >
>>> > Ref: https://issues.apache.org/jira/browse/INFRA-19581
>>> >
>>> > -Ayush
>>> >
>>> > On 23-Dec-2019, at 5:03 PM, Akira Ajisaka  wrote:
>>> >
>>> > Thank you for your work, Jonathan.
>>> >
>>> > I found branch-2 has been unintentionally pushed again. Would you
>>> remove
>>> > it?
>>> > I think the branch should be protected if possible.
>>> >
>>> > -Akira
>>> >
>>> > On Tue, Dec 10, 2019 at 5:17 AM Jonathan Hung 
>>> > wrote:
>>> >
>>> > It's done. The new commit chain is: trunk -> branch-3.2 -> branch-3.1
>>> ->
>>> >
>>> > branch-2.10 -> branch-2.9 -> branch-2.8 (branch-2 no longer exists,
>>> please
>>> >
>>> > don't try to commit to it)
>>> >
>>> >
>>> > Completed procedure:
>>> >
>>> >
>>> >   - Verified everything in old branch-2.10 was in old branch-2
>>> >
>>> >   - Delete old branch-2.10
>>> >
>>> >   - Rename branch-2 to (new) branch-2.10
>>> >
>>> >   - Set version in new branch-2.10 to 2.10.1-SNAPSHOT
>>> >
>>> >   - Renamed fix versions from 2.11.0 to 2.10.1
>>> >
>>> >   - Removed 2.11.0 as a version in HADOOP/YARN/HDFS/MAPREDUCE
>>> >
>>> >
>>> >
>>> > Jonathan Hung
>>> >
>>> >
>>> >
>>> > On Wed, Dec 4, 2019 at 10:55 AM Jonathan Hung 
>>> >
>>> > wrote:
>>> >
>>> >
>>> > FYI, starting the rename process, beginning with INFRA-19521.
>>> >
>>> >
>>> > Jonathan Hung
>>> >
>>> >
>>> >
>>> > On Wed, Nov 27, 2019 at 12:15 PM Konstantin Shvachko <
>>> >
>>> > shv.had...@gmail.com>
>>> >
>>> > wrote:
>>> >
>>> >
>>> > Hey guys,
>>> >
>>> >
>>> > I think we diverged a bit from the initial topic of this discussion,
>>> >
>>> > which is removing branch-2.10, and changing the version of branch-2
>>> from
>>> >
>>> > 2.11.0-SNAPSHOT to 2.10.1-SNAPSHOT.
>>> >
>>> > Sounds like the subject line for this thread "Making 2.10 the last
>>> minor
>>> >
>>> > 2.x release" confused people.
>>> >
>>> > It is in fact a wider matter that can be discussed when somebody
>>> >
>>> > actually
>>> >
>>> > proposes to release 2.11, which I understand nobody does at the moment.
>>> >
>>> >
>>> > So if anybody objects removing branch-2.10 please make an argument.
>>> >
>>> > Otherwise we should go ahead and just do it next week.
>>> >
>>> > I see people still struggling to keep branch-2 and branch-2.10 in sync.
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > --Konstantin
>>> >
>>> >
>>> > On Thu, Nov 21, 2019 at 3:49 PM Jonathan Hung 
>>> >
>>>

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Akira Ajisaka]

Hi Brahma,

How is this issue going?
If there are some blockers, I can help you.

-Akira

On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula 
wrote:

> Hi All,
>
> branch-3.3 created[1] and trunk  set to 3.4.0-SNAPSHOT.
>
> while committing jira's please set the proper fix version and if there is
> any blocker/critical please commit to branch-3.3 or let me know.
>
> Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous
> communication in another thread[2]).
>
>
> 1.https://github.com/apache/hadoop/commits/branch-3.3
> 2.https://lists.apache.org/list.html?common-...@hadoop.apache.org:2020-3
>
>
>
>
> --Brahma Reddy Battula
>

Re: [DISCUSS] Shade guava into hadoop-thirdparty

[Akira Ajisaka]

+1

Thanks,
Akira

On Sun, Apr 5, 2020 at 4:13 AM Wei-Chiu Chuang  wrote:

> Hi Hadoop devs,
>
> I spent a good part of the past 7 months working with a dozen of colleagues
> to update the guava version in Cloudera's software (that includes Hadoop,
> HBase, Spark, Hive, Cloudera Manager ... more than 20+ projects)
>
> After 7 months, I finally came to a conclusion: Update to Hadoop 3.3 /
> 3.2.1 / 3.1.3, even if you just go from Hadoop 3.0/ 3.1.0 is going to be
> really hard because of guava. Because of Guava, the amount of work to
> certify a minor release update is almost equivalent to a major release
> update.
>
> That is because:
> (1) Going from guava 11 to guava 27 is a big jump. There are several
> incompatible API changes in many places. Too bad the Google developers are
> not sympathetic about its users.
> (2) guava is used in all Hadoop jars. Not just Hadoop servers but also
> client jars and Hadoop common libs.
> (3) The Hadoop library is used in practically all software at Cloudera.
>
> Here is my proposal:
> (1) shade guava into hadoop-thirdparty, relocate the classpath to
> org.hadoop.thirdparty.com.google.common.*
> (2) make a hadoop-thirdparty 1.1.0 release.
> (3) update existing references to guava to the relocated path. There are
> more than 2k imports that need an update.
> (4) release Hadoop 3.3.1 / 3.2.2 that contains this change.
>
> In this way, we will be able to update guava in Hadoop in the future
> without disrupting Hadoop applications.
>
> Note: HBase already did this and this guava update project would have been
> much more difficult if HBase didn't do so.
>
> Thoughts? Other options include
> (1) force downstream applications to migrate to Hadoop client artifacts as
> listed here
>
> https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/DownstreamDev.html
> but
> that's nearly impossible.
> (2) Migrate Guava to Java APIs. I suppose this is a big project and I can't
> estimate how much work it's going to be.
>
> Weichiu
>

[jira] [Created] (MAPREDUCE-7269) TestNetworkedJob fails

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7269:


 Summary: TestNetworkedJob fails
 Key: MAPREDUCE-7269
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7269
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1460/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
{noformat}
[INFO] Running org.apache.hadoop.mapred.TestNetworkedJob
[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 20.981 
s <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
[ERROR] testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob)  Time 
elapsed: 4.588 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<[]default> but was:<[root.]default>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Akira Ajisaka]

+1

- Verified signatures and checksums
- Built jars and docs from source
- Built hadoop trunk with hadoop-thirdparty 1.0.0
- Checked rat files and documents
- Checked LICENSE and NOTICE files

Thanks,
Akira

On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B 
wrote:

> Hi folks,
>
> Thanks to everyone's help on this release.
>
> I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty
> 1.0.0.
>
> RC Release artifacts are available at :
>
> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
>
> Maven artifacts are available in staging repo:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>
> The RC tag in git is here:
> https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
>
> And my public key is at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm IST.*
>
> For the testing, I have verified Hadoop trunk compilation with
>"-DdistMgmtSnapshotsUrl=
> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>  -Dhadoop-thirdparty-protobuf.version=1.0.0"
>
> My +1 to start.
>
> -Vinay
>

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Akira Ajisaka]

If you can provide ARM release for future releases, I'm fine with that.

Thanks,
Akira

On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula 
wrote:

> thanks Akira.
>
> Currently only problem is dedicated ARM for future RM.This i want to sort
> out like below,if you've some other,please let me know.
>
> i) Single machine and share cred to future RM ( as we can delete keys once
> release is over).
> ii) Creating the jenkins project ( may be we need to discuss in the
> board..)
> iii) I can provide ARM release for future releases.
>
>
>
>
>
>
>
> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka  wrote:
>
> > Hi Brahma,
> >
> > I think we cannot do any of your proposed actions.
> >
> >
> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> > > Strictly speaking, releases must be verified on hardware owned and
> > controlled by the committer. That means hardware the committer has
> physical
> > possession and control of and exclusively full administrative/superuser
> > access to. That's because only such hardware is qualified to hold a PGP
> > private key, and the release should be verified on the machine the
> private
> > key lives on or on a machine as trusted as that.
> >
> > https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> > > Private keys MUST NOT be stored on any ASF machine. Likewise,
> signatures
> > for releases MUST NOT be created on ASF machines.
> >
> > We need to have dedicated physical ARM machines for each release manager,
> > and now it is not feasible.
> > If you provide an unofficial ARM binary release in some repository,
> that's
> > okay.
> >
> > -Akira
> >
> > On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula 
> > wrote:
> >
> >> Hello folks,
> >>
> >> As currently trunk will support ARM based compilation and qbt(1) is
> >> running
> >> from several months with quite stable, hence planning to propose ARM
> >> binary
> >> this time.
> >>
> >> ( Note : As we'll know voting will be based on the source,so this will
> not
> >> issue.)
> >>
> >> *Proposed Change:*
> >> Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
> >> binary also.?
> >>
> >> *Actions:*
> >> a) *Dedicated* *Machine*:
> >>i) Dedicated ARM machine will be donated which I confirmed
> >>ii) Or can use jenkins ARM machine itself which is currently used
> >> for ARM
> >> b) *Automate Release:* How about having one release project in
> jenkins..?
> >> So that future RM's just trigger the jenkin project.
> >>
> >> Please let me know your thoughts on this.
> >>
> >>
> >> 1.
> >>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> >> 2.https://hadoop.apache.org/releases.html
> >>
> >>
> >>
> >>
> >>
> >>
> >> --Brahma Reddy Battula
> >>
> >
>
> --
>
>
>
> --Brahma Reddy Battula
>

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Akira Ajisaka]

Hi Brahma,

I think we cannot do any of your proposed actions.

http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
> Strictly speaking, releases must be verified on hardware owned and
controlled by the committer. That means hardware the committer has physical
possession and control of and exclusively full administrative/superuser
access to. That's because only such hardware is qualified to hold a PGP
private key, and the release should be verified on the machine the private
key lives on or on a machine as trusted as that.

https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> Private keys MUST NOT be stored on any ASF machine. Likewise, signatures
for releases MUST NOT be created on ASF machines.

We need to have dedicated physical ARM machines for each release manager,
and now it is not feasible.
If you provide an unofficial ARM binary release in some repository, that's
okay.

-Akira

On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula 
wrote:

> Hello folks,
>
> As currently trunk will support ARM based compilation and qbt(1) is running
> from several months with quite stable, hence planning to propose ARM binary
> this time.
>
> ( Note : As we'll know voting will be based on the source,so this will not
> issue.)
>
> *Proposed Change:*
> Currently in downloads we are keeping only x86 binary(2),Can we keep ARM
> binary also.?
>
> *Actions:*
> a) *Dedicated* *Machine*:
>i) Dedicated ARM machine will be donated which I confirmed
>ii) Or can use jenkins ARM machine itself which is currently used
> for ARM
> b) *Automate Release:* How about having one release project in jenkins..?
> So that future RM's just trigger the jenkin project.
>
> Please let me know your thoughts on this.
>
>
> 1.
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
> 2.https://hadoop.apache.org/releases.html
>
>
>
>
>
>
> --Brahma Reddy Battula
>

Re: This week's Hadoop storage community call

[Akira Ajisaka]

+cc: yarn-dev/mapreduce-dev since it contains the information for Hadoop
3.3.0 release

Thank you all for the discussion and thanks Wei-Chiu for the summary.

> I created a dashboard to track the Hadoop 3.3.0 release:
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335311
> It is public to any one, and editable for committers.

This dashboard looks very nice. Thanks.

> Nanda raised a question about how would future release managers access ARM
> machines in order to make ARM release bits.

Making ARM release seems difficult for ASF official release because both
physical and admin access is required.
http://www.apache.org/legal/release-policy.html#owned-controlled-hardware

Regards,
Akira


On Fri, Mar 6, 2020 at 3:38 AM Wei-Chiu Chuang 
wrote:

> Summary:
>
> I created a dashboard to track the Hadoop 3.3.0 release:
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335311
> It is public to any one, and editable for committers.
>
> We discussed the current progress. 3.3.0 is looking good. Brahma is
> planning to cut a branch by the end of next week (correct me if I got it
> wrong).
> Additionally, ARM build is running successfully these days and Brahma wants
> to put up an ARM version of release bits in Hadoop 3.3.0
> Nanda raised a question about how would future release managers access ARM
> machines in order to make ARM release bits.
>
> We also discussed the branch-2.9 EOL discussion. Brahma suggested to invite
> user@ mailing list to join the discussion.
>
> On Mon, Mar 2, 2020 at 1:59 PM Wei-Chiu Chuang  wrote:
>
> > Hi!
> >
> > I'd like to use this week's community call as an opportunity to drive the
> > releases forward. RMs: Gabor and Brahma, would you be able to join the
> call?
> >
> > March 4th (Wednesday) US Pacific: 10am, GMT 6pm, India: 11:30pm
> >
> > Please join via Zoom:
> > https://cloudera.zoom.us/j/880548968
> >
> > Past meeting minutes
> >
> >
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit
> >
> >
>

Re: [VOTE] EOL Hadoop branch-2.8

[Akira Ajisaka]

+1

-Akira

On Tue, Mar 3, 2020 at 4:55 AM Ayush Saxena  wrote:

> +1 for marking 2.8 EOL
>
> -Ayush
>
> > On 03-Mar-2020, at 12:18 AM, Wei-Chiu Chuang  wrote:
> >
> > I am sorry I forgot to start a VOTE thread.
> >
> > This is the "official" vote thread to mark branch-2.8 End of Life. This
> is
> > based on the following thread and the tracking jira (HADOOP-16880
> > <https://issues.apache.org/jira/browse/HADOOP-16880>).
> >
> > This vote will run for 7 days and conclude on March 9th (Mon) 11am PST.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Weichiu
> >
> >> On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang 
> >> wrote:
> >>
> >> Looking at the EOL policy wiki:
> >>
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches
> >>
> >> The Hadoop community can still elect to make security update for EOL'ed
> >> releases.
> >>
> >> I think the EOL is to give more clarity to downstream applications (such
> >> as HBase) the guidance of which Hadoop release lines are still active.
> >> Additionally, I don't think it is sustainable to maintain 6 concurrent
> >> release lines in this big project, which is why I wanted to start this
> >> discussion.
> >>
> >> Thoughts?
> >>
> >>> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan 
> wrote:
> >>>
> >>> Hi Wei-Chiu
> >>>
> >>> Extremely sorry for the late reply here.
> >>> Cud u pls help to add more clarity on defining what will happen for
> >>> branch-2.8 when we call EOL.
> >>> Does this mean that, no more release coming out from this branch, or
> some
> >>> more additional guidelines?
> >>>
> >>> - Sunil
> >>>
> >>>
> >>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
> >>>  wrote:
> >>>
> >>>> This thread has been running for 7 days and no -1.
> >>>>
> >>>> Don't think we've established a formal EOL process, but to publicize
> the
> >>>> EOL, I am going to file a jira, update the wiki and post the
> >>> announcement
> >>>> to general@ and user@
> >>>>
> >>>> On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia <
> >>> dineshc@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Thanks Wei-Chiu for initiating this.
> >>>>>
> >>>>> +1 for 2.8 EOL.
> >>>>>
> >>>>> On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka 
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks Wei-Chiu for starting the discussion,
> >>>>>>
> >>>>>> +1 for the EoL.
> >>>>>>
> >>>>>> -Akira
> >>>>>>
> >>>>>> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena 
> >>>> wrote:
> >>>>>>
> >>>>>>> Thanx Wei-Chiu for initiating this
> >>>>>>> +1 for marking 2.8 EOL
> >>>>>>>
> >>>>>>> -Ayush
> >>>>>>>
> >>>>>>>> On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang <
> >>> weic...@apache.org>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th
> >>>> 2018.
> >>>>>>>>
> >>>>>>>> It's been 17 months since the release and the community by and
> >>>> large
> >>>>>> have
> >>>>>>>> moved up to 2.9/2.10/3.x.
> >>>>>>>>
> >>>>>>>> With Hadoop 3.3.0 over the horizon, is it time to start the EOL
> >>>>>>> discussion
> >>>>>>>> and reduce the number of active branches?
> >>>>>>>
> >>>>>>>
> >>> -
> >>>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>>>>>> For additional commands, e-mail:
> >>> common-dev-h...@hadoop.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] hadoop-thirdparty 1.0.0 release

[Akira Ajisaka]

Thanks Vinayakumar for starting the discussion,

+1 for the release plan.
I think the release vote timeframe is now 5 days, not 7 days.

-Akira

On Fri, Feb 21, 2020 at 3:56 PM Vinayakumar B 
wrote:

> Hi All,
>
> Since Hadoop-3.3.0 release is around the corner, its time to release
> hadoop-thirdparty's first ever release, without which hadoop-3.3.0 cannot
> proceed for release.
>
> Below are the tentative date for RC and release. Since there is no much
> activity in this repo (other than the opentracing related one, which I just
> merged), Keeping the plan little aggressive.
> Please let me know any concerns regarding the same.
>
>  RC-0 : 25-Feb-2020
> Release : 03-Mar-2020 (after 7 days Voting)
>
> -Vinay
>

Re: [DISCUSS] EOL Hadoop branch-2.8

[Akira Ajisaka]

Thanks Wei-Chiu for starting the discussion,

+1 for the EoL.

-Akira

On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena  wrote:

> Thanx Wei-Chiu for initiating this
> +1 for marking 2.8 EOL
>
> -Ayush
>
> > On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang  wrote:
> >
> > The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th 2018.
> >
> > It's been 17 months since the release and the community by and large have
> > moved up to 2.9/2.10/3.x.
> >
> > With Hadoop 3.3.0 over the horizon, is it time to start the EOL
> discussion
> > and reduce the number of active branches?
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Feature branch for HDFS-14978 In-place Erasure Coding Conversion

[Akira Ajisaka]

+1 for the feature branch.

On Fri, Jan 24, 2020 at 6:34 AM Wei-Chiu Chuang
 wrote:

> Hi we are working on a feature to improve Erasure Coding, and I would like
> to seek your opinion on creating a feature branch for it. (HDFS-14978
> )
>
> Reason for a feature branch
> (1) it turns out we need to update NameNode layout version
> (2) It's a medium size project and we want to get this feature merged in
> its entirety.
>
> Aravindan Vijayan and I are planning to work on this feature.
>
> Thoughts?
>

Re: [DISCUSS] Removal of old hamlet package in 3.3+ or 3.4+

[Akira Ajisaka]

Hi folks,

Now I have a strong reason to remove the old hamlet package.

There are 1000+ javadoc warnings in the package and they fill the output of
the precommit javadoc module.
That's why the new warnings and errors are ignored and sometimes they cause
build errors.

Now I'm investigating why the precommit job ignores new javadoc
warnings/errors
in https://issues.apache.org/jira/browse/HADOOP-16802. I'd like to remove
the package to make the investigation easier.

Regards,
Akira

On Thu, Feb 14, 2019 at 3:18 PM Akira Ajisaka  wrote:

> Thanks Masatake for your comments.
> Added the other Hadoop mailing lists to Cc.
>
> > I'm +1 on making imcompatible change if this blocks another Java
> migration issues, while I don't see strong reason to hurry as I see the
> patch of YARN-9279.
> Agreed.
>
> -Akira
>
> On Sun, Feb 10, 2019 at 2:00 AM Masatake Iwasaki
>  wrote:
> >
> > Thanks for working on this, Akira.
> >
> >  > The only usage I can see is Apache Slider, however, the
> >  > functionalities of Apache Slider have been merged into YARN.
> >
> > Do we have mailing lists other than yarn-dev to reach downstream
> developers?
> > It would be better to make it confident that old hamlet of Hadoop 3 is
> > used nowhere.
> >
> > I'm +1 on making imcompatible change if this blocks another Java
> > migration issues,
> > while I don't see strong reason to hurry as I see the patch of YARN-9279.
> >
> > Masatake Iwasaki
> >
> > On 2/3/19 18:10, Akira Ajisaka wrote:
> > > Filed https://issues.apache.org/jira/browse/YARN-9279 to remove the
> > > old hamlet package.
> > >
> > > -Akira
> > >
> > > 2019年1月21日(月) 13:08 Akira Ajisaka :
> > >> Hi folks,
> > >>
> > >> I'd like to remove the deprecated hamlet package to reduce the
> maintenance cost.
> > >>
> > >> The old hamlet package has one character '_' and it is banned in Java
> > >> 9+, so HADOOP-11875 deprecated this package and created a profile in
> > >> pom.xml not to compile the package when the Java version is 9+. After
> > >> the deprecation, we still have to maintenance the profile (see
> > >> YARN-8123 and HADOOP-16046).
> > >>
> > >> The only usage I can see is Apache Slider, however, the
> > >> functionalities of Apache Slider have been merged into YARN. Therefore
> > >> I think there are no people using Slider with Hadoop 3.1+ and we can
> > >> remove the package in 3.3+.
> > >>
> > >> Any thoughts?
> > >>
> > >> Regards,
> > >> Akira
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>

[jira] [Resolved] (MAPREDUCE-7257) Fix incorrect javadoc format

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-7257.
--
Resolution: Duplicate

Thanks [~iwasakims] for the pointer. Closing this as duplicate.

> Fix incorrect javadoc format
> 
>
> Key: MAPREDUCE-7257
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7257
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation
>    Reporter: Akira Ajisaka
>Priority: Major
>  Labels: newbie
>
> `mvn package -Pdist -DskipTests` fails.
> {noformat}
> [ERROR] 
> /Users/aajisaka/git/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
>  error: bad use of '>'
> [ERROR]* @return true if we have number of samples > kMinimumReads and 
> the record
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7257) Fix incorrect javadoc format

[Akira Ajisaka (Jira)]

Akira Ajisaka created MAPREDUCE-7257:


 Summary: Fix incorrect javadoc format
 Key: MAPREDUCE-7257
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7257
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: documentation
Reporter: Akira Ajisaka


`mvn package -Pdist -DskipTests` fails.
{noformat}
[ERROR] 
/Users/aajisaka/git/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/forecast/SimpleExponentialSmoothing.java:134:
 error: bad use of '>'
[ERROR]* @return true if we have number of samples > kMinimumReads and the 
record
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 2019 Release Planning

[Akira Ajisaka]

>  I am interested on 3.3 release ..will act as RM .will update the wiki as
well..

Thanks Brahma for your reply. I'll help you as co-RM.
We will send announcements (cutting branches, code freeze, and so on) in
another thread.

Thanks,
Akira

On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan  wrote:

> Hi guys,
>
> Thanks for the update and for volunteering to be RM.
>
> I just did a quick check:
> 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
> 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
> 3.3.0 has .. many patches sitting here so we definitely need a release.
>
> If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great.
>
> Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw
> community started to have large prod deployment on 3.2.x, Gabor if you have
> bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4.
>
> Thoughts?
> - Wangda
>
> On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula 
> wrote:
>
>> Thanks Akira for resuming this..
>>
>>  I am interested on 3.3 release ..will act as RM .will update the wiki as
>> well..
>>
>>
>>
>> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota 
>> wrote:
>>
>>> I'm interested in doing a release of hadoop.
>>> The version we need an RM is 3.1.3 right? What's the target date for
>>> that?
>>>
>>> Thanks,
>>> Gabor
>>>
>>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka 
>>> wrote:
>>>
>>> > Thank you Wangda.
>>> >
>>> > Now it's 2020. Let's release Hadoop 3.3.0.
>>> > I created a wiki page for tracking blocker/critical issues for 3.3.0
>>> and
>>> > I'll check the issues in the list.
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
>>> > If you find blocker/critical issues in trunk, please set the target
>>> version
>>> > to 3.3.0 for tracking.
>>> >
>>> > > We still need RM for 3.3.0 and 3.1.3.
>>> > I can work as a release manager for 3.3.0. Is there anyone who wants
>>> to be
>>> > a RM?
>>> >
>>> > Thanks and regards,
>>> > Akira
>>> >
>>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
>>> > wrote:
>>> >
>>> > > Thanks Wangda for bring this up!
>>> > >
>>> > > I ran the submarine 0.2.0 release before with a lot of help from
>>> folks
>>> > > especially Sunil. :D
>>> > > And this time I would like to help to release the 3.1.4. Thanks!
>>> > >
>>> > > BR,
>>> > > Zhankun
>>> > >
>>> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
>>> > >
>>> > > > Hi Wangda,
>>> > > > Thanks for bringing this up!
>>> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade
>>> is a
>>> > > > problem.
>>> > > > Hope commiters watch and review these issues, Thanks
>>> > > > https://issues.apache.org/jira/browse/HDFS-13596
>>> > > > https://issues.apache.org/jira/browse/HDFS-14396
>>> > > >
>>> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
>>> > > >
>>> > > > > Hi all,
>>> > > > >
>>> > > > > Hope this email finds you well
>>> > > > >
>>> > > > > I want to hear your thoughts about what should be the release
>>> plan
>>> > for
>>> > > > > 2019.
>>> > > > >
>>> > > > > In 2018, we released:
>>> > > > > - 1 maintenance release of 2.6
>>> > > > > - 3 maintenance releases of 2.7
>>> > > > > - 3 maintenance releases of 2.8
>>> > > > > - 3 releases of 2.9
>>> > > > > - 4 releases of 3.0
>>> > > > > - 2 releases of 3.1
>>> > > > >
>>> > > > > Total 16 releases in 2018.
>>> > > > >
>>> > > > > In 2019, by far we only have two releases:
>>> > > > > - 1 maintenance release of 3.1
>>> > > > > - 1 minor release of 3.2.
>>> > > > >
>>> > > > > However, the community put a lot of efforts to stabilize
>>> features of
>>> > > > > various release branches.
>>> > > > > There're:
>>> > > > > - 217 fixed patches in 3.1.3 [1]
>>> > > > > - 388 fixed patches in 3.2.1 [2]
>>> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
>>> > > > >
>>> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
>>> do a
>>> > > > minor
>>> > > > > release for 3.3.0.
>>> > > > >
>>> > > > > In addition, I saw community discussion to do a 2.8.6 release for
>>> > > > security
>>> > > > > fixes.
>>> > > > >
>>> > > > > Any other releases? I think there're release plans for Ozone as
>>> well.
>>> > > And
>>> > > > > please add your thoughts.
>>> > > > >
>>> > > > > Volunteers welcome! If you have interests to run a release as
>>> Release
>>> > > > > Manager (or co-Resource Manager), please respond to this email
>>> thread
>>> > > so
>>> > > > we
>>> > > > > can coordinate.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Wangda Tan
>>> > > > >
>>> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>>> Fixed
>>> > > AND
>>> > > > > fixVersion = 3.1.3
>>> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>>> Fixed
>>> > > AND
>>> > > > > fixVersion = 3.2.1
>>> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
>>> Fixed
>>> > > AND
>>> > > > > fixVersion = 3.3.0
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>> --
>>
>>
>>
>> --Brahma Reddy Battula
>>
>

Re: [DISCUSS] Hadoop 2019 Release Planning

[Akira Ajisaka]

Thank you Wangda.

Now it's 2020. Let's release Hadoop 3.3.0.
I created a wiki page for tracking blocker/critical issues for 3.3.0 and
I'll check the issues in the list.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
If you find blocker/critical issues in trunk, please set the target version
to 3.3.0 for tracking.

> We still need RM for 3.3.0 and 3.1.3.
I can work as a release manager for 3.3.0. Is there anyone who wants to be
a RM?

Thanks and regards,
Akira

On Fri, Aug 16, 2019 at 9:28 PM zhankun tang  wrote:

> Thanks Wangda for bring this up!
>
> I ran the submarine 0.2.0 release before with a lot of help from folks
> especially Sunil. :D
> And this time I would like to help to release the 3.1.4. Thanks!
>
> BR,
> Zhankun
>
> Hui Fei 于2019年8月16日 周五下午7:19写道：
>
> > Hi Wangda,
> > Thanks for bringing this up!
> > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is a
> > problem.
> > Hope commiters watch and review these issues, Thanks
> > https://issues.apache.org/jira/browse/HDFS-13596
> > https://issues.apache.org/jira/browse/HDFS-14396
> >
> > Wangda Tan  于2019年8月10日周六 上午10:59写道：
> >
> > > Hi all,
> > >
> > > Hope this email finds you well
> > >
> > > I want to hear your thoughts about what should be the release plan for
> > > 2019.
> > >
> > > In 2018, we released:
> > > - 1 maintenance release of 2.6
> > > - 3 maintenance releases of 2.7
> > > - 3 maintenance releases of 2.8
> > > - 3 releases of 2.9
> > > - 4 releases of 3.0
> > > - 2 releases of 3.1
> > >
> > > Total 16 releases in 2018.
> > >
> > > In 2019, by far we only have two releases:
> > > - 1 maintenance release of 3.1
> > > - 1 minor release of 3.2.
> > >
> > > However, the community put a lot of efforts to stabilize features of
> > > various release branches.
> > > There're:
> > > - 217 fixed patches in 3.1.3 [1]
> > > - 388 fixed patches in 3.2.1 [2]
> > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> > >
> > > I think it is the time to do maintenance releases of 3.1/3.2 and do a
> > minor
> > > release for 3.3.0.
> > >
> > > In addition, I saw community discussion to do a 2.8.6 release for
> > security
> > > fixes.
> > >
> > > Any other releases? I think there're release plans for Ozone as well.
> And
> > > please add your thoughts.
> > >
> > > Volunteers welcome! If you have interests to run a release as Release
> > > Manager (or co-Resource Manager), please respond to this email thread
> so
> > we
> > > can coordinate.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed
> AND
> > > fixVersion = 3.1.3
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed
> AND
> > > fixVersion = 3.2.1
> > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed
> AND
> > > fixVersion = 3.3.0
> > >
> >
>

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Akira Ajisaka]

Thank you, Ayush.

I understand we should keep branch-2 as is, as well as master.

-Akira


On Mon, Dec 23, 2019 at 9:14 PM Ayush Saxena  wrote:

> Hi Akira
> Seems there was an INFRA ticket for that. INFRA-19581,
> But the INFRA people closed as wont do and yes, the branch is protected,
> we can’t delete it directly.
>
> Ref: https://issues.apache.org/jira/browse/INFRA-19581
>
> -Ayush
>
> On 23-Dec-2019, at 5:03 PM, Akira Ajisaka  wrote:
>
> Thank you for your work, Jonathan.
>
> I found branch-2 has been unintentionally pushed again. Would you remove
> it?
> I think the branch should be protected if possible.
>
> -Akira
>
> On Tue, Dec 10, 2019 at 5:17 AM Jonathan Hung 
> wrote:
>
> It's done. The new commit chain is: trunk -> branch-3.2 -> branch-3.1 ->
>
> branch-2.10 -> branch-2.9 -> branch-2.8 (branch-2 no longer exists, please
>
> don't try to commit to it)
>
>
> Completed procedure:
>
>
>   - Verified everything in old branch-2.10 was in old branch-2
>
>   - Delete old branch-2.10
>
>   - Rename branch-2 to (new) branch-2.10
>
>   - Set version in new branch-2.10 to 2.10.1-SNAPSHOT
>
>   - Renamed fix versions from 2.11.0 to 2.10.1
>
>   - Removed 2.11.0 as a version in HADOOP/YARN/HDFS/MAPREDUCE
>
>
>
> Jonathan Hung
>
>
>
> On Wed, Dec 4, 2019 at 10:55 AM Jonathan Hung 
>
> wrote:
>
>
> FYI, starting the rename process, beginning with INFRA-19521.
>
>
> Jonathan Hung
>
>
>
> On Wed, Nov 27, 2019 at 12:15 PM Konstantin Shvachko <
>
> shv.had...@gmail.com>
>
> wrote:
>
>
> Hey guys,
>
>
> I think we diverged a bit from the initial topic of this discussion,
>
> which is removing branch-2.10, and changing the version of branch-2 from
>
> 2.11.0-SNAPSHOT to 2.10.1-SNAPSHOT.
>
> Sounds like the subject line for this thread "Making 2.10 the last minor
>
> 2.x release" confused people.
>
> It is in fact a wider matter that can be discussed when somebody
>
> actually
>
> proposes to release 2.11, which I understand nobody does at the moment.
>
>
> So if anybody objects removing branch-2.10 please make an argument.
>
> Otherwise we should go ahead and just do it next week.
>
> I see people still struggling to keep branch-2 and branch-2.10 in sync.
>
>
> Thanks,
>
> --Konstantin
>
>
> On Thu, Nov 21, 2019 at 3:49 PM Jonathan Hung 
>
> wrote:
>
>
> Thanks for the detailed thoughts, everyone.
>
>
> Eric (Badger), my understanding is the same as yours re. minor vs patch
>
> releases. As for putting features into minor/patch releases, if we
>
> keep the
>
> convention of putting new features only into minor releases, my
>
> assumption
>
> is still that it's unlikely people will want to get them into branch-2
>
> (based on the 2.10.0 release process). For the java 11 issue, we
>
> haven't
>
> even really removed support for java 7 in branch-2 (much less java 8),
>
> so I
>
> feel moving to java 11 would go along with a move to branch 3. And as
>
> you
>
> mentioned, if people really want to use java 11 on branch-2, we can
>
> always
>
> revive branch-2. But for now I think the convenience of not needing to
>
> port
>
> to both branch-2 and branch-2.10 (and below) outweighs the cost of
>
> potentially needing to revive branch-2.
>
>
> Jonathan Hung
>
>
>
> On Wed, Nov 20, 2019 at 10:50 AM Eric Yang  wrote:
>
>
> +1 for 2.10.x as last release for 2.x version.
>
>
> Software would become more compatible when more companies stress test
>
> the same software and making improvements in trunk.  Some may be extra
>
> caution on moving up the version because obligation internally to keep
>
> things running.  Company obligation should not be the driving force to
>
> maintain Hadoop branches.  There is no proper collaboration in the
>
> community when every name brand company maintains its own Hadoop 2.x
>
> version.  I think it would be more healthy for the community to
>
> reduce the
>
> branch forking and spend energy on trunk to harden the software.
>
> This will
>
> give more confidence to move up the version than trying to fix n
>
> permutations breakage like Flash fixing the timeline.
>
>
> Apache license stated, there is no warranty of any kind for code
>
> contributions.  Fewer community release process should improve
>
> software
>
> quality when eyes are on trunk, and help steering toward the same end
>
> goals.
>
>
> regards,
>
> Eric
>
>
>
>
> On Tue, Nov 19, 2019 at 3:03 PM Eric Badger

Re: [DISCUSS] Making 2.10 the last minor 2.x release

[Akira Ajisaka]

Thank you for your work, Jonathan.

I found branch-2 has been unintentionally pushed again. Would you remove it?
I think the branch should be protected if possible.

-Akira

On Tue, Dec 10, 2019 at 5:17 AM Jonathan Hung  wrote:

> It's done. The new commit chain is: trunk -> branch-3.2 -> branch-3.1 ->
> branch-2.10 -> branch-2.9 -> branch-2.8 (branch-2 no longer exists, please
> don't try to commit to it)
>
> Completed procedure:
>
>- Verified everything in old branch-2.10 was in old branch-2
>- Delete old branch-2.10
>- Rename branch-2 to (new) branch-2.10
>- Set version in new branch-2.10 to 2.10.1-SNAPSHOT
>- Renamed fix versions from 2.11.0 to 2.10.1
>- Removed 2.11.0 as a version in HADOOP/YARN/HDFS/MAPREDUCE
>
>
> Jonathan Hung
>
>
> On Wed, Dec 4, 2019 at 10:55 AM Jonathan Hung 
> wrote:
>
> > FYI, starting the rename process, beginning with INFRA-19521.
> >
> > Jonathan Hung
> >
> >
> > On Wed, Nov 27, 2019 at 12:15 PM Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> >> Hey guys,
> >>
> >> I think we diverged a bit from the initial topic of this discussion,
> >> which is removing branch-2.10, and changing the version of branch-2 from
> >> 2.11.0-SNAPSHOT to 2.10.1-SNAPSHOT.
> >> Sounds like the subject line for this thread "Making 2.10 the last minor
> >> 2.x release" confused people.
> >> It is in fact a wider matter that can be discussed when somebody
> actually
> >> proposes to release 2.11, which I understand nobody does at the moment.
> >>
> >> So if anybody objects removing branch-2.10 please make an argument.
> >> Otherwise we should go ahead and just do it next week.
> >> I see people still struggling to keep branch-2 and branch-2.10 in sync.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Thu, Nov 21, 2019 at 3:49 PM Jonathan Hung 
> >> wrote:
> >>
> >>> Thanks for the detailed thoughts, everyone.
> >>>
> >>> Eric (Badger), my understanding is the same as yours re. minor vs patch
> >>> releases. As for putting features into minor/patch releases, if we
> keep the
> >>> convention of putting new features only into minor releases, my
> assumption
> >>> is still that it's unlikely people will want to get them into branch-2
> >>> (based on the 2.10.0 release process). For the java 11 issue, we
> haven't
> >>> even really removed support for java 7 in branch-2 (much less java 8),
> so I
> >>> feel moving to java 11 would go along with a move to branch 3. And as
> you
> >>> mentioned, if people really want to use java 11 on branch-2, we can
> always
> >>> revive branch-2. But for now I think the convenience of not needing to
> port
> >>> to both branch-2 and branch-2.10 (and below) outweighs the cost of
> >>> potentially needing to revive branch-2.
> >>>
> >>> Jonathan Hung
> >>>
> >>>
> >>> On Wed, Nov 20, 2019 at 10:50 AM Eric Yang  wrote:
> >>>
>  +1 for 2.10.x as last release for 2.x version.
> 
>  Software would become more compatible when more companies stress test
>  the same software and making improvements in trunk.  Some may be extra
>  caution on moving up the version because obligation internally to keep
>  things running.  Company obligation should not be the driving force to
>  maintain Hadoop branches.  There is no proper collaboration in the
>  community when every name brand company maintains its own Hadoop 2.x
>  version.  I think it would be more healthy for the community to
> reduce the
>  branch forking and spend energy on trunk to harden the software.
> This will
>  give more confidence to move up the version than trying to fix n
>  permutations breakage like Flash fixing the timeline.
> 
>  Apache license stated, there is no warranty of any kind for code
>  contributions.  Fewer community release process should improve
> software
>  quality when eyes are on trunk, and help steering toward the same end
> goals.
> 
>  regards,
>  Eric
> 
> 
> 
>  On Tue, Nov 19, 2019 at 3:03 PM Eric Badger
>   wrote:
> 
> > Hello all,
> >
> > Is it written anywhere what the difference is between a minor release
> > and a
> > point/dot/maintenance (I'll use "point" from here on out) release? I
> > have
> > looked around and I can't find anything other than some compatibility
> > documentation in 2.x that has since been removed in 3.x [1] [2]. I
> > think
> > this would help shape my opinion on whether or not to keep branch-2
> > alive.
> > My current understanding is that we can't really break compatibility
> in
> > either a minor or point release. But the only mention of the
> difference
> > between minor and point releases is how to deal with Stable,
> Evolving,
> > and
> > Unstable tags, and how to deal with changing default configuration
> > values.
> > So it seems like there really isn't a big official difference between
> > the
> > two. In my mind, the functional 

Re: [DISCUSS] Enable github security notifications to all Hadoop committers

[Akira Ajisaka]

Thanks Wei-Chiu for starting the discussion,

+1

> I already have hundreds of incoming emails
from various Hadoop alias each day so I don't mind adding a few more.

Completely agreed.

-Akira

On Thu, Oct 24, 2019 at 3:48 PM Bharat Viswanadham
 wrote:

> +1 (binding). I am interested in receiving these notifications.
>
>
> Thanks,
> Bharat
>
>
>
> On Wed, Oct 23, 2019 at 11:38 PM Wei-Chiu Chuang 
> wrote:
>
> > Hi,
> > I raised INFRA-19327 
> > to
> > enable github security notification. How do people feel if we enable this
> > notification to all committers? I already have hundreds of incoming
> emails
> > from various Hadoop alias each day so I don't mind adding a few more.
> >
> > If there is no good consensus, then we'll have to let committers to
> opt-in
> > individually.
> >
> > Thanks,
> > Weichiu
> >
>

Re: [DISCUSS] GitHub PRs without JIRA number

[Akira Ajisaka]

Thanks all for the discussion.

I'm +1 for adding PR template and I wrote a patch long ago:
https://issues.apache.org/jira/browse/HADOOP-15184

I'm interested in "test this", "retest this", etc.
I'll file a jira for this feature.

-Akira

On Thu, Sep 5, 2019 at 4:05 AM Sean Busbey 
wrote:

> We should add a Pull Request Template that specifically calls out the
> expectation that folks need to have a JIRA associated with their PR
> for it to get reviewed. Expectations around time to response and how
> to go about getting attention when things lag would also be good to
> include. (e.g. are folks expected to ping on the jira? are folks
> expected to email a relevant *-dev list?)
>
> If anyone is interested in doing the work to make it so "test this" /
> "retest this" / etc work, open a jira and I'll give you some pointers
> of examples to go off of. We use a plugin to do this for yetus based
> tests in some HBase repos.
>
> On Wed, Sep 4, 2019 at 1:59 PM Wei-Chiu Chuang
>  wrote:
> >
> > +general@
> >
> >
> > On Wed, Aug 28, 2019 at 6:42 AM Wei-Chiu Chuang 
> wrote:
> >
> > > I don't think our GitHub integration supports those commands. Ozone has
> > > its own github integration that can test individual PRs though.
> > >
> > >
> > >
> > > On Tue, Aug 27, 2019 at 12:40 PM Iñigo Goiri 
> wrote:
> > >
> > >> I wouldn't go for #3 and always require a JIRA for a PR.
> > >>
> > >> In general, I think we should state the best practices for using
> GitHub
> > >> PRs.
> > >> There were some guidelines but they were kind of open
> > >> For example, adding always a link to the JIRA to the description.
> > >> I think PRs can have a template as a start.
> > >>
> > >> The other thing I would do is to disable the automatic Jenkins
> trigger.
> > >> I've seen the "retest this" and others:
> > >>
> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
> > >> https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md
> > >>
> > >>
> > >>
> > >> On Tue, Aug 27, 2019 at 10:47 AM Wei-Chiu Chuang 
> > >> wrote:
> > >>
> > >> > Hi,
> > >> > There are hundreds of GitHub PRs pending review. Many of them just
> sit
> > >> > there wasting Jenkins resources.
> > >> >
> > >> > I suggest:
> > >> > (1) close PRs that went stale (i.e. doesn't compile). Or even close
> PRs
> > >> > that hasn't been reviewed for more than a year.
> > >> > (1) close PRs that doesn't have a JIRA number. No one is going to
> > >> review a
> > >> > big PR that doesn't have a JIRA anyway.
> > >> > (2) For PRs without JIRA number, file JIRAs for the PR on behalf of
> the
> > >> > reporter.
> > >> > (3) For typo fixes, merge the PRs directly without a JIRA. IMO,
> this is
> > >> the
> > >> > best use of GitHub PR.
> > >> >
> > >> > Thoughts?
> > >> >
> > >>
> > >
>
>
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Akira Ajisaka]

+1

Thanks,
Akira

On Sat, Aug 24, 2019 at 11:06 AM Wangda Tan  wrote:
>
> Hi devs,
>
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
>
> This voting thread will run for 7 days and will end at Aug 30th.
>
> Please let me know if you have any questions.
>
> Thanks,
> Wangda Tan

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Akira Ajisaka]

+1

-Akira

On Wed, Aug 21, 2019 at 12:10 PM Jonathan Hung  wrote:
>
> +1. Thanks!
>
> Jonathan Hung
>
>
> On Tue, Aug 20, 2019 at 8:03 PM Wangda Tan  wrote:
>
> > Hi all,
> >
> > This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> > and 3.0 EOL. This is based on discussions of [1]
> >
> > This discussion runs for 7 days and will conclude on Aug 28 Wed.
> >
> > Please feel free to share your thoughts.
> >
> > Thanks,
> > Wangda
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> > ,
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Make EOL branches to public?

[Akira Ajisaka]

+1

Thank you for the discussion.

-Akira

On Wed, Aug 21, 2019 at 5:51 AM Wei-Chiu Chuang  wrote:
>
> +1
> I feel like one year of inactivity is a good sign that the community is not
> interested in the branch any more.
>
> On Fri, Aug 16, 2019 at 3:14 AM Wangda Tan  wrote:
>
> > Hi folks,
> >
> > Want to hear your thoughts about what we should do to make some branches
> > EOL. We discussed a couple of times before in dev lists and PMC list.
> > However, we couldn't get a formal process of EOL. According to the
> > discussion. It is hard to decide it based on time like "After 1st release,
> > EOL in 2 years". Because community members still want to maintain it and
> > developers still want to get a newer version released.
> >
> > However, without a public place to figure out which release will be EOL, it
> > is very hard for users to choose the right releases to upgrade and develop.
> >
> > So I want to propose to make an agreement about making a public EOL wiki
> > page and create a process to EOL a release:
> >
> > The process I'm thinking is very simple: If no volunteer to do a
> > maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> > year). We will claim a release is EOL. After EOL community can still choose
> > to do a security-only release.
> >
> > Here's a list which I can think about:
> >
> > 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> > 2) 2.7.x (Last released at Apr 2018)
> > 4) 3.0.x (Last released at May 2018)
> >
> > Thoughts?
> >
> > Thanks,
> > Wangda
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 2019 Release Planning

[Akira Ajisaka]

Thanks all for the discussion,

- I'd like the volunteered release managers to review this issue:
https://issues.apache.org/jira/browse/HADOOP-16494
- I can co-RM for a release.

Thanks and regards,
Akira

On Fri, Aug 16, 2019 at 6:49 PM Wangda Tan  wrote:
>
> Thanks all for the suggestions and volunteering run these releases:
>
> I just updated roadmap of Hadoop wiki:
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
>
> Now I put 5 releases and planned date is 2019. We have 3 releases have
> volunteered RM (Release Manager):
>
> - 3.2.1: Rohith
> - 2.8.6: Junping
> - 2.10.0: Jonathan
>
> We still need RM for 3.3.0 and 3.1.3.
>
> Also, from my past experience, it will be helpful to get a co-RM (or shadow
> RM) to do release together since release will have some effort which two
> RMs can share some workload.
>
> Can you help to update Roadmap wiki and put an estimated release date,
> feature freeze date, etc. Which we can starting release cycle sooner if
> possible?
>
> Thanks,
> Wangda
>
>
> On Fri, Aug 16, 2019 at 5:00 PM Rohith Sharma K S 
> wrote:
>
> > Hi Wangda,
> >
> > Thanks for initiating this. I would like to nominate myself for 3.2.1
> > Release Management.
> >
> > -Rohith Sharma K S
> >
> > On Sat, 10 Aug 2019 at 08:29, Wangda Tan  wrote:
> >
> > > Hi all,
> > >
> > > Hope this email finds you well
> > >
> > > I want to hear your thoughts about what should be the release plan for
> > > 2019.
> > >
> > > In 2018, we released:
> > > - 1 maintenance release of 2.6
> > > - 3 maintenance releases of 2.7
> > > - 3 maintenance releases of 2.8
> > > - 3 releases of 2.9
> > > - 4 releases of 3.0
> > > - 2 releases of 3.1
> > >
> > > Total 16 releases in 2018.
> > >
> > > In 2019, by far we only have two releases:
> > > - 1 maintenance release of 3.1
> > > - 1 minor release of 3.2.
> > >
> > > However, the community put a lot of efforts to stabilize features of
> > > various release branches.
> > > There're:
> > > - 217 fixed patches in 3.1.3 [1]
> > > - 388 fixed patches in 3.2.1 [2]
> > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> > >
> > > I think it is the time to do maintenance releases of 3.1/3.2 and do a
> > minor
> > > release for 3.3.0.
> > >
> > > In addition, I saw community discussion to do a 2.8.6 release for
> > security
> > > fixes.
> > >
> > > Any other releases? I think there're release plans for Ozone as well. And
> > > please add your thoughts.
> > >
> > > Volunteers welcome! If you have interests to run a release as Release
> > > Manager (or co-Resource Manager), please respond to this email thread so
> > we
> > > can coordinate.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > > fixVersion = 3.1.3
> > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > > fixVersion = 3.2.1
> > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND
> > > fixVersion = 3.3.0
> > >
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Prefer JUnit5 for new tests

[Akira Ajisaka]

My answer:

JUnit 5 -> org.junit.jupiter.api.Test
JUnit 4 -> org.junit.Test
(JUnit 3 -> junit.framework.TestCase)

I found there are still some tests written in JUnit 3 API...

Sorry for the late reply.

-Akira

On Fri, Jul 26, 2019 at 8:37 PM Steve Loughran  wrote:
>
>
> this is a silly question, but what is a "Junit 5 test"?
>
> We've been slowly adopting the assertJ APIs in new tests in hadoop-aws, and 
> they work file in the older codebase, so even for existing tests we can 
> advocate them. they're very good for making assertions about collections; 
> very verbose for classic assertTrue/assertFalse, but can be used to generate 
> great error strings
>
> On Fri, Jul 26, 2019 at 9:26 AM Akira Ajisaka  wrote:
>>
>> Hi folks,
>>
>> Now we are slowly migrating from JUnit4 to JUnit5.
>> https://issues.apache.org/jira/browse/HADOOP-14693
>>
>> However, as Steve commented [1], if we are going to migrate the
>> existing tests, the backporting cost will become too expensive.
>> Therefore, I'd like to recommend using JUnit5 for new tests before
>> migrating the existing tests. Using junit-vintage-engine, we can mix
>> JUnit4 and JUnit5 APIs in the same module, so writing new tests in
>> JUnit5 is relatively easy.
>>
>> Any thoughts?
>>
>> [1] 
>> https://issues.apache.org/jira/browse/HADOOP-16318?focusedCommentId=16890955=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16890955
>>
>> -Akira
>>
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Resolved] (MAPREDUCE-6973) Fix comments on creating _SUCCESS file.

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved MAPREDUCE-6973.
--
Resolution: Fixed

Committed this to trunk. Thanks [~mehulgarnara].

> Fix comments on creating _SUCCESS file.
> ---
>
> Key: MAPREDUCE-6973
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6973
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.0.0-beta1
>Reporter: Mehul Garnara (MG)
>Assignee: Mehul Garnara (MG)
>Priority: Trivial
>  Labels: easyfix, newbie
> Fix For: 3.3.0
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> I went through couple of old JIRA issues and understood that earlier app was 
> creating "_done" file on job has completed successfully. After some 
> conversation by group decided to create "_SUCCESS" instead of "_done". 
> However, while learning the code, I found there is one comment has reference 
> of "_done" and would like to start with small contribution to fix it. 
> Note: I would like to work on this trivial issue so can get opportunity to 
> follow standard process of contribution steps, that will myself to come on 
> track quickly for future contribution that I would like to do. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] Prefer JUnit5 for new tests

[Akira Ajisaka]

Hi folks,

Now we are slowly migrating from JUnit4 to JUnit5.
https://issues.apache.org/jira/browse/HADOOP-14693

However, as Steve commented [1], if we are going to migrate the
existing tests, the backporting cost will become too expensive.
Therefore, I'd like to recommend using JUnit5 for new tests before
migrating the existing tests. Using junit-vintage-engine, we can mix
JUnit4 and JUnit5 APIs in the same module, so writing new tests in
JUnit5 is relatively easy.

Any thoughts?

[1] 
https://issues.apache.org/jira/browse/HADOOP-16318?focusedCommentId=16890955=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16890955

-Akira

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org