Re: [DISCUSS] Hadoop 3.3.1 release
Hi Bilwa, I have commented on the jira's you mentioned. Based on the stability we can plan this.But needs to be merged ASAP. On Fri, Feb 19, 2021 at 5:20 PM bilwa st wrote: > Hi Brahma, > > Can we have below features in 3.3.1 release? We have been using these > features for a long time. They are stable and tested in bigger clusters. > > 1. Container reuse - https://issues.apache.org/jira/browse/MAPREDUCE-6749 > 2. Speculative attempts should not run on the same node - > https://issues.apache.org/jira/browse/MAPREDUCE-7169 > > Thanks, > Bilwa > > On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula > wrote: > >> Sorry for the late reply.. >> >> I will come up with a plan.. Please let me know if anybody has some >> features/improvements/bugs that need to be included. >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan wrote: >> >> > Hi Wei-Chiu, >> > >> > What will be the next steps here for 3.3.1 planning? >> > >> > Thanks >> > Sunil >> > >> > On Mon, Feb 8, 2021 at 11:56 PM Stack wrote: >> > >> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran >> > > > > > >> > > wrote: >> > > >> > > > >> > > > Regarding blockers : how about we have a little hackathon where we >> > try >> > > > and get things in. This means a promise of review time from the >> people >> > > with >> > > > commit rights and other people who understand the code (Stack?) >> > > > >> > > > >> > > >> > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing). >> > > Thanks, >> > > S >> > > >> > > >> > > >> > > >> > > > -steve >> > > > >> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena >> wrote: >> > > > >> > > > > +1 >> > > > > Just to mention we would need to release hadoop-thirdparty too >> > before. >> > > > > Presently we are using the snapshot version of it. >> > > > > >> > > > > -Ayush >> > > > > >> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang > > >> > > > wrote: >> > > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've >> > > > > accumulated >> > > > > > more than 400 changes in the branch-3.3. A number of >> downstreamers >> > > are >> > > > > > eagerly waiting for 3.3.1 which addresses the guava version >> > conflict >> > > > > issue. >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20 >> > > > > > >> > > > > > We should start the release work for 3.3.1 before the diff >> becomes >> > > even >> > > > > > larger. >> > > > > > >> > > > > > I believe there are currently only two real blockers for a >> 3.3.1 >> > > > (using >> > > > > > this filter >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC >> > > > > > ) >> > > > > > >> > > > > > >> > > > > > 1. HDFS-15566 < >> https://issues.apache.org/jira/browse/HDFS-15566> >> > > > > > 2. >> > > > > > 1. HADOOP-17112 < >> > > > https://issues.apache.org/jira/browse/HADOOP-17112 >> > > > > > >> > > > > > 2. >> > > > > > >> > > > > > >> > > > > > >> > > > > > Is there anyone who would volunteer to be the 3.3.1 RM? >> > > > > > >> > > > > > Also, the HowToRelease wiki does not describe the ARM build >> > process. >> > > > > That's >> > > > > > going to be important for future releases. >> > > > > >> > > > > >> - >> > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > > > >> > > > > >> > > > >> > > >> > >> >> >> -- >> >> >> >> --Brahma Reddy Battula >> > -- --Brahma Reddy Battula
Re: Seeing YETUS issue in jenkins runs
Hi Steve and Ahmed, Thank you for taking care of this issue. > take care: first it's spotbugs, next it'll be maven, before long you'll be > doing linux device drivers... I hope I won't be doing linux device drivers... > HADOOP-16870 Sorry, I've updated the configuration and now it works again: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/426/ Akira On Tue, Feb 23, 2021 at 6:02 AM Ahmed Hussein wrote: > > I noticed that the qbt-reports were not generated for the last 2 days. > It is probably related to 16870. I filed a jira > https://issues.apache.org/jira/browse/HADOOP-17541 > > > On Mon, Feb 22, 2021 at 1:26 PM Steve Loughran > wrote: > > > Akira, > > > > I see you've started fixing upstream OSS projects. > > > > take care: first it's spotbugs, next it'll be maven, before long you'll be > > doing linux device drivers... > > > > > > > > On Fri, 19 Feb 2021 at 05:53, Akira Ajisaka wrote: > > > > > Probably this issue is due to > > > https://issues.apache.org/jira/browse/HADOOP-16870. Reverted. > > > > > > -Akira > > > > > > On Fri, Feb 19, 2021 at 11:27 AM Sunil Govindan > > wrote: > > > > > > > > Hi All, > > > > > > > > We are getting the below error in recent runs. > > > > > > > > -1. yetus. 0m 8s. Unprocessed flag(s): --findbugs-strict-precheck > > > > > > > > https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console > > > > > > > > Are there any knows issues or some workarounds? > > > > Please advise. > > > > > > > > Thanks > > > > Sunil > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > > > > -- > Best Regards, > > *Ahmed Hussein, PhD* - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Seeing YETUS issue in jenkins runs
I noticed that the qbt-reports were not generated for the last 2 days. It is probably related to 16870. I filed a jira https://issues.apache.org/jira/browse/HADOOP-17541 On Mon, Feb 22, 2021 at 1:26 PM Steve Loughran wrote: > Akira, > > I see you've started fixing upstream OSS projects. > > take care: first it's spotbugs, next it'll be maven, before long you'll be > doing linux device drivers... > > > > On Fri, 19 Feb 2021 at 05:53, Akira Ajisaka wrote: > > > Probably this issue is due to > > https://issues.apache.org/jira/browse/HADOOP-16870. Reverted. > > > > -Akira > > > > On Fri, Feb 19, 2021 at 11:27 AM Sunil Govindan > wrote: > > > > > > Hi All, > > > > > > We are getting the below error in recent runs. > > > > > > -1. yetus. 0m 8s. Unprocessed flag(s): --findbugs-strict-precheck > > > > > > https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console > > > > > > Are there any knows issues or some workarounds? > > > Please advise. > > > > > > Thanks > > > Sunil > > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > -- Best Regards, *Ahmed Hussein, PhD*
Re: Seeing YETUS issue in jenkins runs
Akira, I see you've started fixing upstream OSS projects. take care: first it's spotbugs, next it'll be maven, before long you'll be doing linux device drivers... On Fri, 19 Feb 2021 at 05:53, Akira Ajisaka wrote: > Probably this issue is due to > https://issues.apache.org/jira/browse/HADOOP-16870. Reverted. > > -Akira > > On Fri, Feb 19, 2021 at 11:27 AM Sunil Govindan wrote: > > > > Hi All, > > > > We are getting the below error in recent runs. > > > > -1. yetus. 0m 8s. Unprocessed flag(s): --findbugs-strict-precheck > > > > https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console > > > > Are there any knows issues or some workarounds? > > Please advise. > > > > Thanks > > Sunil > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (HADOOP-17541) Yetus does not run qbt-trunk
Ahmed Hussein created HADOOP-17541:
--
Summary: Yetus does not run qbt-trunk
Key: HADOOP-17541
URL: https://issues.apache.org/jira/browse/HADOOP-17541
Project: Hadoop Common
Issue Type: Bug
Components: bin, build, yetus
Reporter: Ahmed Hussein
On Feb20th, qbt-reports started to generate empty reports
{code:bash}
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/424/
ERROR: File 'out/email-report.txt' does not exist
{code}
On Jenkins, the job fails with the following error:
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/425/console
{code:bash}
ERROR:
/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64//dev-support/bin/hadoop.sh
does not exist.
Build step 'Execute shell' marked build as failure
Archiving artifacts
[Fast Archiver] No prior successful build to compare, so performing full copy
of artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
{code}
[~aajisaka], I think this would be caused by HADOOP-16748 . I noticed that the
PR of that HADOOP-16748 ceased from showing any reports, but for some reason I
forgot about that while reviewing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17540) hadoop-client-runtime-3.3.0 not properly shading javax.xml.bind
Lorie created HADOOP-17540: -- Summary: hadoop-client-runtime-3.3.0 not properly shading javax.xml.bind Key: HADOOP-17540 URL: https://issues.apache.org/jira/browse/HADOOP-17540 Project: Hadoop Common Issue Type: Bug Components: hdfs-client Affects Versions: 3.3.0 Reporter: Lorie In the file hadoop-client-modules/hadoop-client-runtime/pom.xml, it excludes javax.xml.bind because it says it is part of the SE JDK. But, this is no longer true in Java 11. So, when I try to use Guice to inject objects at start up, I am getting the error Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/xmlbind/ModuleUtil (wrong name: javax/xml/bind/ModuleUtil) || | || | || | |javax.xml.bind| | |jaxb-api| | || I was able to get past this by using zip to change the position of the ModuleUtil.class inside the META-INF directory. Before I made any changes: jar tvf hadoop-client-runtime-3.3.0.jar | grep ModuleUtil* META-INF/versions/9/javax/xml/bind/ModuleUtil.class org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class After I modified the jar: jar tvf hadoop-client-runtime-3.3.0.jar | grep ModuleUtil* META-INF/versions/9/org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class org/apache/hadoop/shaded/javax/xml/bind/ModuleUtil.class With the revised jar file, my code started. I don't know if that is the correct way to fix but it works. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17539) ABFS test setup failing "The specified filesystem already exists"
Steve Loughran created HADOOP-17539:
---
Summary: ABFS test setup failing "The specified filesystem already
exists"
Key: HADOOP-17539
URL: https://issues.apache.org/jira/browse/HADOOP-17539
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure, test
Affects Versions: 3.3.1
Reporter: Steve Loughran
been hitting this in parallel test runs. Even though FS Creation is meant to
catch and swallow exceptions of code
{{AzureServiceErrorCode.FILE_SYSTEM_ALREADY_EXISTS}}, this doesn't happen
because the exception is translated into FileAlreadyExistsException before that
catch/swallow
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/ No changes -1 overall The following subsystems voted -1: asflicense hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-tools/hadoop-azure/src/config/checkstyle.xml hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml Failed junit tests : hadoop.util.TestDiskCheckerWithDiskIo hadoop.fs.TestFileUtil hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.namenode.TestEditLogRace hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.web.TestWebHDFS hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler hadoop.yarn.server.resourcemanager.TestRMProxyUsersConf hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.yarn.sls.appmaster.TestAMSimulator hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-compile-javac-root.txt [456K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-patch-pylint.txt [60K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/whitespace-tabs.txt [1.3M] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/xml.txt [4.0K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/diff-javadoc-javadoc-root.txt [20K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [216K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [276K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/216/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K]
[jira] [Created] (HADOOP-17538) Add kms-default.xml and httpfs-default.xml to site index
Masatake Iwasaki created HADOOP-17538: - Summary: Add kms-default.xml and httpfs-default.xml to site index Key: HADOOP-17538 URL: https://issues.apache.org/jira/browse/HADOOP-17538 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki While there are links to *-default.xml in the site index, kms-default.xml and httpfs-default.xml are not included. Adding them could be useful as a quick reference. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17537) Correct assertion reversed in HADOOP-13327
Sumangala Patki created HADOOP-17537: Summary: Correct assertion reversed in HADOOP-13327 Key: HADOOP-17537 URL: https://issues.apache.org/jira/browse/HADOOP-17537 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.3.0 Reporter: Sumangala Patki HADOOP-13327 introduces, among other changes, functions to simplify assert for checking stream capabilities. This PR fixes (originally) assertFalse statements whose logic has been reversed to assertTrue when the above change was checked-in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
