Re: [DISCUSS] Hadoop 2019 Release Planning
> I am interested on 3.3 release ..will act as RM .will update the wiki as well.. Thanks Brahma for your reply. I'll help you as co-RM. We will send announcements (cutting branches, code freeze, and so on) in another thread. Thanks, Akira On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan wrote: > Hi guys, > > Thanks for the update and for volunteering to be RM. > > I just did a quick check: > 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21) > 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22) > 3.3.0 has .. many patches sitting here so we definitely need a release. > > If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great. > > Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw > community started to have large prod deployment on 3.2.x, Gabor if you have > bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4. > > Thoughts? > - Wangda > > On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula > wrote: > >> Thanks Akira for resuming this.. >> >> I am interested on 3.3 release ..will act as RM .will update the wiki as >> well.. >> >> >> >> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota >> wrote: >> >>> I'm interested in doing a release of hadoop. >>> The version we need an RM is 3.1.3 right? What's the target date for >>> that? >>> >>> Thanks, >>> Gabor >>> >>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka >>> wrote: >>> >>> > Thank you Wangda. >>> > >>> > Now it's 2020. Let's release Hadoop 3.3.0. >>> > I created a wiki page for tracking blocker/critical issues for 3.3.0 >>> and >>> > I'll check the issues in the list. >>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release >>> > If you find blocker/critical issues in trunk, please set the target >>> version >>> > to 3.3.0 for tracking. >>> > >>> > > We still need RM for 3.3.0 and 3.1.3. >>> > I can work as a release manager for 3.3.0. Is there anyone who wants >>> to be >>> > a RM? >>> > >>> > Thanks and regards, >>> > Akira >>> > >>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang >>> > wrote: >>> > >>> > > Thanks Wangda for bring this up! >>> > > >>> > > I ran the submarine 0.2.0 release before with a lot of help from >>> folks >>> > > especially Sunil. :D >>> > > And this time I would like to help to release the 3.1.4. Thanks! >>> > > >>> > > BR, >>> > > Zhankun >>> > > >>> > > Hui Fei 于2019年8月16日 周五下午7:19写道： >>> > > >>> > > > Hi Wangda, >>> > > > Thanks for bringing this up! >>> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade >>> is a >>> > > > problem. >>> > > > Hope commiters watch and review these issues, Thanks >>> > > > https://issues.apache.org/jira/browse/HDFS-13596 >>> > > > https://issues.apache.org/jira/browse/HDFS-14396 >>> > > > >>> > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： >>> > > > >>> > > > > Hi all, >>> > > > > >>> > > > > Hope this email finds you well >>> > > > > >>> > > > > I want to hear your thoughts about what should be the release >>> plan >>> > for >>> > > > > 2019. >>> > > > > >>> > > > > In 2018, we released: >>> > > > > - 1 maintenance release of 2.6 >>> > > > > - 3 maintenance releases of 2.7 >>> > > > > - 3 maintenance releases of 2.8 >>> > > > > - 3 releases of 2.9 >>> > > > > - 4 releases of 3.0 >>> > > > > - 2 releases of 3.1 >>> > > > > >>> > > > > Total 16 releases in 2018. >>> > > > > >>> > > > > In 2019, by far we only have two releases: >>> > > > > - 1 maintenance release of 3.1 >>> > > > > - 1 minor release of 3.2. >>> > > > > >>> > > > > However, the community put a lot of efforts to stabilize >>> features of >>> > > > > various release branches. >>> > > > > There're: >>> > > > > - 217 fixed patches in 3.1.3 [1] >>> > > > > - 388 fixed patches in 3.2.1 [2] >>> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) >>> > > > > >>> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and >>> do a >>> > > > minor >>> > > > > release for 3.3.0. >>> > > > > >>> > > > > In addition, I saw community discussion to do a 2.8.6 release for >>> > > > security >>> > > > > fixes. >>> > > > > >>> > > > > Any other releases? I think there're release plans for Ozone as >>> well. >>> > > And >>> > > > > please add your thoughts. >>> > > > > >>> > > > > Volunteers welcome! If you have interests to run a release as >>> Release >>> > > > > Manager (or co-Resource Manager), please respond to this email >>> thread >>> > > so >>> > > > we >>> > > > > can coordinate. >>> > > > > >>> > > > > Thanks, >>> > > > > Wangda Tan >>> > > > > >>> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >>> Fixed >>> > > AND >>> > > > > fixVersion = 3.1.3 >>> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >>> Fixed >>> > > AND >>> > > > > fixVersion = 3.2.1 >>> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >>> Fixed >>> > > AND >>> > > > > fixVersion = 3.3.0 >>> > > > > >>> > > > >>> > > >>> > >>> >> -- >> >> >> >> --Brahma Reddy Battula >> >
Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/560/ No changes - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [DISCUSS] Hadoop 2019 Release Planning
Hi guys, Thanks for the update and for volunteering to be RM. I just did a quick check: 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21) 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22) 3.3.0 has .. many patches sitting here so we definitely need a release. If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great. Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw community started to have large prod deployment on 3.2.x, Gabor if you have bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4. Thoughts? - Wangda On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula wrote: > Thanks Akira for resuming this.. > > I am interested on 3.3 release ..will act as RM .will update the wiki as > well.. > > > > On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota > wrote: > >> I'm interested in doing a release of hadoop. >> The version we need an RM is 3.1.3 right? What's the target date for that? >> >> Thanks, >> Gabor >> >> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka wrote: >> >> > Thank you Wangda. >> > >> > Now it's 2020. Let's release Hadoop 3.3.0. >> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and >> > I'll check the issues in the list. >> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release >> > If you find blocker/critical issues in trunk, please set the target >> version >> > to 3.3.0 for tracking. >> > >> > > We still need RM for 3.3.0 and 3.1.3. >> > I can work as a release manager for 3.3.0. Is there anyone who wants to >> be >> > a RM? >> > >> > Thanks and regards, >> > Akira >> > >> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang >> > wrote: >> > >> > > Thanks Wangda for bring this up! >> > > >> > > I ran the submarine 0.2.0 release before with a lot of help from folks >> > > especially Sunil. :D >> > > And this time I would like to help to release the 3.1.4. Thanks! >> > > >> > > BR, >> > > Zhankun >> > > >> > > Hui Fei 于2019年8月16日 周五下午7:19写道： >> > > >> > > > Hi Wangda, >> > > > Thanks for bringing this up! >> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade >> is a >> > > > problem. >> > > > Hope commiters watch and review these issues, Thanks >> > > > https://issues.apache.org/jira/browse/HDFS-13596 >> > > > https://issues.apache.org/jira/browse/HDFS-14396 >> > > > >> > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： >> > > > >> > > > > Hi all, >> > > > > >> > > > > Hope this email finds you well >> > > > > >> > > > > I want to hear your thoughts about what should be the release plan >> > for >> > > > > 2019. >> > > > > >> > > > > In 2018, we released: >> > > > > - 1 maintenance release of 2.6 >> > > > > - 3 maintenance releases of 2.7 >> > > > > - 3 maintenance releases of 2.8 >> > > > > - 3 releases of 2.9 >> > > > > - 4 releases of 3.0 >> > > > > - 2 releases of 3.1 >> > > > > >> > > > > Total 16 releases in 2018. >> > > > > >> > > > > In 2019, by far we only have two releases: >> > > > > - 1 maintenance release of 3.1 >> > > > > - 1 minor release of 3.2. >> > > > > >> > > > > However, the community put a lot of efforts to stabilize features >> of >> > > > > various release branches. >> > > > > There're: >> > > > > - 217 fixed patches in 3.1.3 [1] >> > > > > - 388 fixed patches in 3.2.1 [2] >> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) >> > > > > >> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and >> do a >> > > > minor >> > > > > release for 3.3.0. >> > > > > >> > > > > In addition, I saw community discussion to do a 2.8.6 release for >> > > > security >> > > > > fixes. >> > > > > >> > > > > Any other releases? I think there're release plans for Ozone as >> well. >> > > And >> > > > > please add your thoughts. >> > > > > >> > > > > Volunteers welcome! If you have interests to run a release as >> Release >> > > > > Manager (or co-Resource Manager), please respond to this email >> thread >> > > so >> > > > we >> > > > > can coordinate. >> > > > > >> > > > > Thanks, >> > > > > Wangda Tan >> > > > > >> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >> Fixed >> > > AND >> > > > > fixVersion = 3.1.3 >> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >> Fixed >> > > AND >> > > > > fixVersion = 3.2.1 >> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = >> Fixed >> > > AND >> > > > > fixVersion = 3.3.0 >> > > > > >> > > > >> > > >> > >> > -- > > > > --Brahma Reddy Battula >
Re: [DISCUSS] drop mapreduce-pipes
As long as there is a proper announcement and deprecation path I'm +1. I know some third party products relied on hadoop pipes as a workaround to run parallelized version of their existing binaries, so assuming some are still being used people should be aware of this proposal. Thanks! esteban. -- Cloudera, Inc. On Mon, Jan 6, 2020 at 8:52 AM Brahma Reddy Battula wrote: > +1 on deleting this . > > > On Mon, 6 Jan 2020 at 7:39 PM, Steve Loughran > > wrote: > > > It has been many years since anyone did any work on mapreduce-pipes and > > given the large number of hadoop-cluster streaming frameworks in the ASF > > alone, probably the same number of years since anyone actually launched a > > pipe job other than during testing. > > > > As such, it's a background noise maintenance problem and extra delays in > > test runs. > > > > What do people think about deleting it? > > > > -Steve > > > -- > > > > --Brahma Reddy Battula >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1373/ No changes -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core Class org.apache.hadoop.applications.mawo.server.common.TaskStatus implements Cloneable but does not define or use clone method At TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 39-346] Equals method for org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument is of type WorkerId At WorkerId.java:the argument is of type WorkerId At WorkerId.java:[line 114] org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does not check for null argument At WorkerId.java:null argument At WorkerId.java:[lines 114-115] FindBugs : module:hadoop-cloud-storage-project/hadoop-cos Redundant nullcheck of dir, which is known to be non-null in org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at BufferPool.java:is known to be non-null in org.apache.hadoop.fs.cosn.BufferPool.createDir(String) Redundant null check at BufferPool.java:[line 66] org.apache.hadoop.fs.cosn.CosNInputStream$ReadBuffer.getBuffer() may expose internal representation by returning CosNInputStream$ReadBuffer.buffer At CosNInputStream.java:by returning CosNInputStream$ReadBuffer.buffer At CosNInputStream.java:[line 87] Found reliance on default encoding in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, byte[]):in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFile(String, File, byte[]): new String(byte[]) At CosNativeFileSystemStore.java:[line 199] Found reliance on default encoding in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, InputStream, byte[], long):in org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.storeFileWithRetry(String, InputStream, byte[], long): new String(byte[]) At CosNativeFileSystemStore.java:[line 178] org.apache.hadoop.fs.cosn.CosNativeFileSystemStore.uploadPart(File, String, String, int) may fail to clean up java.io.InputStream Obligation to clean up resource created at CosNativeFileSystemStore.java:fail to clean up java.io.InputStream Obligation to clean up resource created at CosNativeFileSystemStore.java:[line 252] is not discharged Failed junit tests : hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.hdfs.server.namenode.TestRedudantBlocks hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown hadoop.yarn.applications.distributedshell.TestDistributedShell
Re: [DISCUSS] drop mapreduce-pipes
+1 on deleting this . On Mon, 6 Jan 2020 at 7:39 PM, Steve Loughran wrote: > It has been many years since anyone did any work on mapreduce-pipes and > given the large number of hadoop-cluster streaming frameworks in the ASF > alone, probably the same number of years since anyone actually launched a > pipe job other than during testing. > > As such, it's a background noise maintenance problem and extra delays in > test runs. > > What do people think about deleting it? > > -Steve > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 2019 Release Planning
Thanks Akira for resuming this.. I am interested on 3.3 release ..will act as RM .will update the wiki as well.. On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota wrote: > I'm interested in doing a release of hadoop. > The version we need an RM is 3.1.3 right? What's the target date for that? > > Thanks, > Gabor > > On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka wrote: > > > Thank you Wangda. > > > > Now it's 2020. Let's release Hadoop 3.3.0. > > I created a wiki page for tracking blocker/critical issues for 3.3.0 and > > I'll check the issues in the list. > > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release > > If you find blocker/critical issues in trunk, please set the target > version > > to 3.3.0 for tracking. > > > > > We still need RM for 3.3.0 and 3.1.3. > > I can work as a release manager for 3.3.0. Is there anyone who wants to > be > > a RM? > > > > Thanks and regards, > > Akira > > > > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang > > wrote: > > > > > Thanks Wangda for bring this up! > > > > > > I ran the submarine 0.2.0 release before with a lot of help from folks > > > especially Sunil. :D > > > And this time I would like to help to release the 3.1.4. Thanks! > > > > > > BR, > > > Zhankun > > > > > > Hui Fei 于2019年8月16日 周五下午7:19写道： > > > > > > > Hi Wangda, > > > > Thanks for bringing this up! > > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is > a > > > > problem. > > > > Hope commiters watch and review these issues, Thanks > > > > https://issues.apache.org/jira/browse/HDFS-13596 > > > > https://issues.apache.org/jira/browse/HDFS-14396 > > > > > > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： > > > > > > > > > Hi all, > > > > > > > > > > Hope this email finds you well > > > > > > > > > > I want to hear your thoughts about what should be the release plan > > for > > > > > 2019. > > > > > > > > > > In 2018, we released: > > > > > - 1 maintenance release of 2.6 > > > > > - 3 maintenance releases of 2.7 > > > > > - 3 maintenance releases of 2.8 > > > > > - 3 releases of 2.9 > > > > > - 4 releases of 3.0 > > > > > - 2 releases of 3.1 > > > > > > > > > > Total 16 releases in 2018. > > > > > > > > > > In 2019, by far we only have two releases: > > > > > - 1 maintenance release of 3.1 > > > > > - 1 minor release of 3.2. > > > > > > > > > > However, the community put a lot of efforts to stabilize features > of > > > > > various release branches. > > > > > There're: > > > > > - 217 fixed patches in 3.1.3 [1] > > > > > - 388 fixed patches in 3.2.1 [2] > > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) > > > > > > > > > > I think it is the time to do maintenance releases of 3.1/3.2 and > do a > > > > minor > > > > > release for 3.3.0. > > > > > > > > > > In addition, I saw community discussion to do a 2.8.6 release for > > > > security > > > > > fixes. > > > > > > > > > > Any other releases? I think there're release plans for Ozone as > well. > > > And > > > > > please add your thoughts. > > > > > > > > > > Volunteers welcome! If you have interests to run a release as > Release > > > > > Manager (or co-Resource Manager), please respond to this email > thread > > > so > > > > we > > > > > can coordinate. > > > > > > > > > > Thanks, > > > > > Wangda Tan > > > > > > > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.1.3 > > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.2.1 > > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.3.0 > > > > > > > > > > > > > > > -- --Brahma Reddy Battula
Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.util.TestReadWriteDiskValidator hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-compile-cc-root-jdk1.8.0_232.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-compile-javac-root-jdk1.8.0_232.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/whitespace-tabs.txt [1.3M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_232.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [232K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/559/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt [12K]
[DISCUSS] drop mapreduce-pipes
It has been many years since anyone did any work on mapreduce-pipes and given the large number of hadoop-cluster streaming frameworks in the ASF alone, probably the same number of years since anyone actually launched a pipe job other than during testing. As such, it's a background noise maintenance problem and extra delays in test runs. What do people think about deleting it? -Steve
Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts
Hi Sree, > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating Project ? Or as a TLP ? > Or as a new project definition ? As already mentioned by Ayush, this will be a subproject of Hadoop. Releases will be voted by Hadoop PMC as per ASF process. > The effort to streamline and put in an accepted standard for the dependencies that require shading, > seems beyond the siloed efforts of hadoop, hbase, etc >I propose, we bring all the decision makers from all these artifacts in one room and decide best course of action. > I am looking at, no projects should ever had to shade any artifacts except as an absolute necessary alternative. This is the ideal proposal for any project. But unfortunately some projects takes their own course based on need. In the current case of protobuf in Hadoop, Protobuf upgrade from 2.5.0 (which is already EOL) was not taken up to avoid downstream failures. Since Hadoop is a platform, its dependencies will get added to downstream projects' classpath. So any change in Hadoop's dependencies will directly affect downstreams. Hadoop strictly follows backward compatibility as far as possible. Though protobuf provides wire compatibility b/w versions, it doesnt provide compatibility for generated sources. Now, to support ARM protobuf upgrade is mandatory. Using shading technique, In Hadoop internally can upgrade to shaded protobuf 3.x and still have 2.5.0 protobuf (deprecated) for downstreams. This shading is necessary to have both versions of protobuf supported. (2.5.0 (non-shaded) for downstream's classpath and 3.x (shaded) for hadoop's internal usage). And this entire work to be done before 3.3.0 release. So, though its ideal to make a common approach for all projects, I suggest for Hadoop we can go ahead as per current approach. We can also start the parallel effort to address these problems in a separate discussion/proposal. Once the solution is available we can revisit and adopt new solution accordingly in all such projects (ex: HBase, Hadoop, Ratis). -Vinay On Mon, Jan 6, 2020 at 12:39 AM Ayush Saxena wrote: > Hey Sree > > > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating > > Project ? Or as a TLP ? > > Or as a new project definition ? > > > A sub project of Apache Hadoop, having its own independent release cycles. > May be you can put this into the same column as ozone or as > submarine(couple of months ago). > > Unifying for all, seems interesting but each project is independent and has > its own limitations and way of thinking, I don't think it would be an easy > task to bring all on the same table and get them agree to a common stuff. > > I guess this has been into discussion since quite long, and there hasn't > been any other alternative suggested. Still we can hold up for a week, if > someone comes up with a better solution, else we can continue in the > present direction. > > -Ayush > > > > On Sun, 5 Jan 2020 at 05:03, Sree Vaddi > wrote: > > > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating > > Project ? Or as a TLP ? > > Or as a new project definition ? > > > > The effort to streamline and put in an accepted standard for the > > dependencies that require shading,seems beyond the siloed efforts of > > hadoop, hbase, etc > > > > I propose, we bring all the decision makers from all these artifacts in > > one room and decide best course of action.I am looking at, no projects > > should ever had to shade any artifacts except as an absolute necessary > > alternative. > > > > > > Thank you./Sree > > > > > > > > On Saturday, January 4, 2020, 7:49:18 AM PST, Vinayakumar B < > > vinayakum...@apache.org> wrote: > > > > Hi, > > Sorry for the late reply,. > > >>> To be exact, how can we better use the thirdparty repo? Looking at > > HBase as an example, it looks like everything that are known to break a > lot > > after an update get shaded into the hbase-thirdparty artifact: guava, > > netty, ... etc. > > Is it the purpose to isolate these naughty dependencies? > > Yes, shading is to isolate these naughty dependencies from downstream > > classpath and have independent control on these upgrades without breaking > > downstreams. > > > > First PR https://github.com/apache/hadoop-thirdparty/pull/1 to create > the > > protobuf shaded jar is ready to merge. > > > > Please take a look if anyone interested, will be merged may be after two > > days if no objections. > > > > -Vinay > > > > > > On Thu, Oct 10, 2019 at 3:30 AM Wei-Chiu Chuang > > wrote: > > > > > Hi I am late to this but I am keen to understand more. > > > > > > To be exact, how can we better use the thirdparty repo? Looking at > HBase > > > as an example, it looks like everything that are known to break a lot > > after > > > an update get shaded into the hbase-thirdparty artifact: guava, netty, > > ... > > > etc. > > > Is it the purpose to isolate these naughty dependencies? > > > > > > On Wed, Oct 9, 2019 at 12:38 PM Vinayakumar
Re: [DISCUSS] Hadoop 2019 Release Planning
I'm interested in doing a release of hadoop. The version we need an RM is 3.1.3 right? What's the target date for that? Thanks, Gabor On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka wrote: > Thank you Wangda. > > Now it's 2020. Let's release Hadoop 3.3.0. > I created a wiki page for tracking blocker/critical issues for 3.3.0 and > I'll check the issues in the list. > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release > If you find blocker/critical issues in trunk, please set the target version > to 3.3.0 for tracking. > > > We still need RM for 3.3.0 and 3.1.3. > I can work as a release manager for 3.3.0. Is there anyone who wants to be > a RM? > > Thanks and regards, > Akira > > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang > wrote: > > > Thanks Wangda for bring this up! > > > > I ran the submarine 0.2.0 release before with a lot of help from folks > > especially Sunil. :D > > And this time I would like to help to release the 3.1.4. Thanks! > > > > BR, > > Zhankun > > > > Hui Fei 于2019年8月16日 周五下午7:19写道： > > > > > Hi Wangda, > > > Thanks for bringing this up! > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is a > > > problem. > > > Hope commiters watch and review these issues, Thanks > > > https://issues.apache.org/jira/browse/HDFS-13596 > > > https://issues.apache.org/jira/browse/HDFS-14396 > > > > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： > > > > > > > Hi all, > > > > > > > > Hope this email finds you well > > > > > > > > I want to hear your thoughts about what should be the release plan > for > > > > 2019. > > > > > > > > In 2018, we released: > > > > - 1 maintenance release of 2.6 > > > > - 3 maintenance releases of 2.7 > > > > - 3 maintenance releases of 2.8 > > > > - 3 releases of 2.9 > > > > - 4 releases of 3.0 > > > > - 2 releases of 3.1 > > > > > > > > Total 16 releases in 2018. > > > > > > > > In 2019, by far we only have two releases: > > > > - 1 maintenance release of 3.1 > > > > - 1 minor release of 3.2. > > > > > > > > However, the community put a lot of efforts to stabilize features of > > > > various release branches. > > > > There're: > > > > - 217 fixed patches in 3.1.3 [1] > > > > - 388 fixed patches in 3.2.1 [2] > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) > > > > > > > > I think it is the time to do maintenance releases of 3.1/3.2 and do a > > > minor > > > > release for 3.3.0. > > > > > > > > In addition, I saw community discussion to do a 2.8.6 release for > > > security > > > > fixes. > > > > > > > > Any other releases? I think there're release plans for Ozone as well. > > And > > > > please add your thoughts. > > > > > > > > Volunteers welcome! If you have interests to run a release as Release > > > > Manager (or co-Resource Manager), please respond to this email thread > > so > > > we > > > > can coordinate. > > > > > > > > Thanks, > > > > Wangda Tan > > > > > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.1.3 > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.2.1 > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed > > AND > > > > fixVersion = 3.3.0 > > > > > > > > > >
