Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 from me too. The vote passed, so I'll continue with the rest of the release. Thanks everyone! Jonathan Hung On Tue, Oct 29, 2019 at 1:40 PM Giovanni Matteo Fumarola < giovanni.fumar...@gmail.com> wrote: > +1 (non-binding). > > - Built from source on Ubuntu with OpenJDK 11.0.3 > - Verified signatures > - Verified documentation > - Setup up a single node cluster and ran basic yarn commands > - Ran UTs for Yarn Router, Yarn Common, Yarn API, YARN NM and YARN RM. > > Thanks for putting this together, Jonathan. > > On Tue, Oct 29, 2019 at 8:47 AM Dinesh Chitlangia > wrote: > >> +1 (non-binding) >> >> - Verified signatures >> - Verified documentation >> - Built from sources on CentOS 7 >> - Tested with basic hdfs commands on a single node setup. >> >> Thank for organizing the release, Jonathan. >> >> -Dinesh >> >> >> >> On Tue, Oct 29, 2019 at 9:45 AM epa...@apache.org >> wrote: >> >> > Compatibility testing has gone well for me. >> > >> > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.8.5 >> and >> > 2.10.0 >> > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.10.0 >> and >> > trunk >> > - With one 4-node cluster running 2.10.0 and one 4-node cluster running >> > trunk, I ran a word count job in each cluster whose inputs and outputs >> were >> > from and to the opposite cluster. >> > - I verified that HDFS replication works as expected in a trunk cluster >> > that has one 2.10.0 datanode. >> > >> > Thanks, >> > -Eric >> > >> > >> > > On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung < >> > jyhung2...@gmail.com> wrote: >> > > Hi folks, >> > > >> > >This is the second release candidate for the first release of Apache >> > Hadoop >> > >2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It >> includes >> > >features such as: >> > > >> > > - User-defined resource types >> > > - Native GPU support as a schedulable resource type >> > > - Consistent reads from standby node >> > > - Namenode port based selective encryption >> > > - Improvements related to rolling upgrade support from 2.x to 3.x >> > > - Cost based fair call queue >> > > >> > > The RC1 artifacts are at: >> > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ >> > > >> > > RC tag is release-2.10.0-RC1. >> > > >> > > The maven artifacts are hosted here: >> > > >> https://repository.apache.org/content/repositories/orgapachehadoop-1243/ >> > > >> > > My public key is available here: >> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> > > >> > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm >> > PDT. >> > > >> > > Thanks, >> > > Jonathan Hung >> > >> > >> > - >> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > >> > >> >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 (non-binding). - Built from source on Ubuntu with OpenJDK 11.0.3 - Verified signatures - Verified documentation - Setup up a single node cluster and ran basic yarn commands - Ran UTs for Yarn Router, Yarn Common, Yarn API, YARN NM and YARN RM. Thanks for putting this together, Jonathan. On Tue, Oct 29, 2019 at 8:47 AM Dinesh Chitlangia wrote: > +1 (non-binding) > > - Verified signatures > - Verified documentation > - Built from sources on CentOS 7 > - Tested with basic hdfs commands on a single node setup. > > Thank for organizing the release, Jonathan. > > -Dinesh > > > > On Tue, Oct 29, 2019 at 9:45 AM epa...@apache.org > wrote: > > > Compatibility testing has gone well for me. > > > > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.8.5 > and > > 2.10.0 > > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.10.0 > and > > trunk > > - With one 4-node cluster running 2.10.0 and one 4-node cluster running > > trunk, I ran a word count job in each cluster whose inputs and outputs > were > > from and to the opposite cluster. > > - I verified that HDFS replication works as expected in a trunk cluster > > that has one 2.10.0 datanode. > > > > Thanks, > > -Eric > > > > > > > On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung < > > jyhung2...@gmail.com> wrote: > > > Hi folks, > > > > > >This is the second release candidate for the first release of Apache > > Hadoop > > >2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > > >features such as: > > > > > > - User-defined resource types > > > - Native GPU support as a schedulable resource type > > > - Consistent reads from standby node > > > - Namenode port based selective encryption > > > - Improvements related to rolling upgrade support from 2.x to 3.x > > > - Cost based fair call queue > > > > > > The RC1 artifacts are at: > > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > > > > > RC tag is release-2.10.0-RC1. > > > > > > The maven artifacts are hosted here: > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > > > > > My public key is available here: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > > PDT. > > > > > > Thanks, > > > Jonathan Hung > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 (non-binding) - Verified signatures - Verified documentation - Built from sources on CentOS 7 - Tested with basic hdfs commands on a single node setup. Thank for organizing the release, Jonathan. -Dinesh On Tue, Oct 29, 2019 at 9:45 AM epa...@apache.org wrote: > Compatibility testing has gone well for me. > > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.8.5 and > 2.10.0 > - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.10.0 and > trunk > - With one 4-node cluster running 2.10.0 and one 4-node cluster running > trunk, I ran a word count job in each cluster whose inputs and outputs were > from and to the opposite cluster. > - I verified that HDFS replication works as expected in a trunk cluster > that has one 2.10.0 datanode. > > Thanks, > -Eric > > > > On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung < > jyhung2...@gmail.com> wrote: > > Hi folks, > > > >This is the second release candidate for the first release of Apache > Hadoop > >2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > >features such as: > > > > - User-defined resource types > > - Native GPU support as a schedulable resource type > > - Consistent reads from standby node > > - Namenode port based selective encryption > > - Improvements related to rolling upgrade support from 2.x to 3.x > > - Cost based fair call queue > > > > The RC1 artifacts are at: > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > > > RC tag is release-2.10.0-RC1. > > > > The maven artifacts are hosted here: > > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > > > My public key is available here: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > PDT. > > > > Thanks, > > Jonathan Hung > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Compatibility testing has gone well for me. - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.8.5 and 2.10.0 - In a 4-node cluster, I ran YARN rolling upgrade tests between 2.10.0 and trunk - With one 4-node cluster running 2.10.0 and one 4-node cluster running trunk, I ran a word count job in each cluster whose inputs and outputs were from and to the opposite cluster. - I verified that HDFS replication works as expected in a trunk cluster that has one 2.10.0 datanode. Thanks, -Eric > On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung > wrote: > Hi folks, > >This is the second release candidate for the first release of Apache Hadoop >2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes >features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > My public key is available here: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 on RC1 - Verified signatures - Built from source (RHEL7) - Ran single node cluster and tested basic HDFS commands Thanks for putting this together Jonathan! On Mon, Oct 28, 2019 at 7:54 PM zhankun tang wrote: > Thanks for the efforts! Jonathan! > > +1 (non-binding) on RC1. > - Set up a single node cluster with the binary tarball > - Run Spark Pi and PySpark job > > BR, > Zhankun > > On Mon, 28 Oct 2019 at 23:16, Masatake Iwasaki < > iwasak...@oss.nttdata.co.jp> > wrote: > > > Thanks for putting this up, Jonathan Hung. > > > > +1(non-binding) > > > > * verified signature and checksum. > > * built source tarball by OpenJDK 7 on CentOS 7 with native profile > > enabled. > > * launched pseudo distributed cluster with security configurations. > > * create encryption zone and put files in it. > > * accessed files in encryption zone via httpfs. > > * checked that file names containing '%' or '+' or ';' can be handled by > > webhdfs. > > > > Masatake Iwasaki > > > > On 10/23/19 06:55, Jonathan Hung wrote: > > > Hi folks, > > > > > > This is the second release candidate for the first release of Apache > > Hadoop > > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It > includes > > > features such as: > > > > > > - User-defined resource types > > > - Native GPU support as a schedulable resource type > > > - Consistent reads from standby node > > > - Namenode port based selective encryption > > > - Improvements related to rolling upgrade support from 2.x to 3.x > > > - Cost based fair call queue > > > > > > The RC1 artifacts are at: > > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > > > > > RC tag is release-2.10.0-RC1. > > > > > > The maven artifacts are hosted here: > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > > > > > My public key is available here: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > > PDT. > > > > > > Thanks, > > > Jonathan Hung > > > > > > [1] > > > > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0) > > > > > > > > > - > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Thanks for the efforts! Jonathan! +1 (non-binding) on RC1. - Set up a single node cluster with the binary tarball - Run Spark Pi and PySpark job BR, Zhankun On Mon, 28 Oct 2019 at 23:16, Masatake Iwasaki wrote: > Thanks for putting this up, Jonathan Hung. > > +1(non-binding) > > * verified signature and checksum. > * built source tarball by OpenJDK 7 on CentOS 7 with native profile > enabled. > * launched pseudo distributed cluster with security configurations. > * create encryption zone and put files in it. > * accessed files in encryption zone via httpfs. > * checked that file names containing '%' or '+' or ';' can be handled by > webhdfs. > > Masatake Iwasaki > > On 10/23/19 06:55, Jonathan Hung wrote: > > Hi folks, > > > > This is the second release candidate for the first release of Apache > Hadoop > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > > features such as: > > > > - User-defined resource types > > - Native GPU support as a schedulable resource type > > - Consistent reads from standby node > > - Namenode port based selective encryption > > - Improvements related to rolling upgrade support from 2.x to 3.x > > - Cost based fair call queue > > > > The RC1 artifacts are at: > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > > > RC tag is release-2.10.0-RC1. > > > > The maven artifacts are hosted here: > > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > > > My public key is available here: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > PDT. > > > > Thanks, > > Jonathan Hung > > > > [1] > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0) > > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Sounds good to me, thank you for addressing the concerns Jonathan. +1 from me. From: Masatake Iwasaki Sent: Monday, October 28, 2019 7:52 AM To: Jonathan Hung ; Hdfs-dev ; mapreduce-dev ; yarn-dev ; Hadoop Common Subject: Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1) Thanks for putting this up, Jonathan Hung. +1(non-binding) * verified signature and checksum. * built source tarball by OpenJDK 7 on CentOS 7 with native profile enabled. * launched pseudo distributed cluster with security configurations. * create encryption zone and put files in it. * accessed files in encryption zone via httpfs. * checked that file names containing '%' or '+' or ';' can be handled by webhdfs. Masatake Iwasaki On 10/23/19 06:55, Jonathan Hung wrote: > Hi folks, > > This is the second release candidate for the first release of Apache Hadoop > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: > https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fhome.apache.org%2F~jhung%2Fhadoop-2.10.0-RC1%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C0b3ef8c0869e47cc242308d75bb9df45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637078726244394622&sdata=S2lETuGnJ7JSZy6A4n7J3vEXpzFvkOf0kxTBy%2Ft%2BO1I%3D&reserved=0 > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1243%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C0b3ef8c0869e47cc242308d75bb9df45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637078726244394622&sdata=FxWR3ecDwSibT98VH8sHOCmcaiV8y8NqIOgmNSyxnCs%3D&reserved=0 > > My public key is available here: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Frelease%2Fhadoop%2Fcommon%2FKEYS&data=02%7C01%7Cekrogen%40linkedin.com%7C0b3ef8c0869e47cc242308d75bb9df45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637078726244404621&sdata=Rgx9nT9QK3X%2Fm1UlsUcJGCe2FON28LY02RJ2hEfhklE%3D&reserved=0 > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung > > [1] > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fissues%2F%3Fjql%3Dproject%2520in%2520(HDFS%252C%2520YARN%252C%2520HADOOP%252C%2520MAPREDUCE)%2520AND%2520resolution%2520%253D%2520Fixed%2520AND%2520fixVersion%2520%253D%25202.10.0%2520AND%2520fixVersion%2520not%2520in%2520(2.9.2%252C%25202.9.1%252C%25202.9.0&data=02%7C01%7Cekrogen%40linkedin.com%7C0b3ef8c0869e47cc242308d75bb9df45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637078726244404621&sdata=LJGuxpenXwsMxY%2BUiWf2rI5Mw2SRuPSkaYHLVxVrnNY%3D&reserved=0) > - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Thanks for putting this up, Jonathan Hung. +1(non-binding) * verified signature and checksum. * built source tarball by OpenJDK 7 on CentOS 7 with native profile enabled. * launched pseudo distributed cluster with security configurations. * create encryption zone and put files in it. * accessed files in encryption zone via httpfs. * checked that file names containing '%' or '+' or ';' can be handled by webhdfs. Masatake Iwasaki On 10/23/19 06:55, Jonathan Hung wrote: Hi folks, This is the second release candidate for the first release of Apache Hadoop 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes features such as: - User-defined resource types - Native GPU support as a schedulable resource type - Consistent reads from standby node - Namenode port based selective encryption - Improvements related to rolling upgrade support from 2.x to 3.x - Cost based fair call queue The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ RC tag is release-2.10.0-RC1. The maven artifacts are hosted here: https://repository.apache.org/content/repositories/orgapachehadoop-1243/ My public key is available here: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. Thanks, Jonathan Hung [1] https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Some more thoughts: for the javadoc issue, I think we can just support building on java 7. For the release notes issue, I can work with the authors of the major features to come up with release notes and update them before pushing it to site. The release notes in the published artifacts won't be up to date, but I think that's fine. I'll go ahead with this plan if no objections. Jonathan Hung On Fri, Oct 25, 2019 at 12:19 PM Jonathan Hung wrote: > Thanks for looking Erik. > > For the release notes, yeah I think it's because there's no release notes > for the corresponding JIRAs. I've added details for these features to the > index.md.vm file which should show up on the homepage for 2.10.0 (e.g. > https://hadoop.apache.org/docs/r2.9.0/index.html). We could add release > notes for these JIRAs, but that would require recreating the tar.gzs since > the release notes are bundled in there. > > For the javadoc issue, I was able to repro this issue, seems it's because > the org.apache.hadoop.yarn.client.ClientRMProxy import was removed in > FederationProxyProviderUtil in YARN-7900 in branch-2 (but not in other > branches). But it's referenced in javadocs in this file so it throws this > error. Re-adding this import and building with java 8 allows it to succeed. > > I checked javadoc html for FederationProxyProviderUtil in the produced > artifacts and it appears to be correct. > > I think we could easily overwrite the current RC1 artifacts with ones > containing proper release notes. Not sure what to do about the javadoc > issue though, that would require overwriting the release-2.10.0-RC1 tag > which I don't want to do. What do others think? > > Jonathan Hung > > > On Fri, Oct 25, 2019 at 9:21 AM Erik Krogen wrote: > >> Thanks for putting this together, Jonathan! >> >> I noticed that the RELEASENOTES.md makes no mention of any of the major >> features you mentioned in your email about the RC. Is this expected? I >> guess it is caused by the lack of a release note on the JIRAs for those >> features. >> >> I also noticed that building a distribution package (mvn -DskipTests >> package -Pdist) fails on Java 8 (1.8.0_172) with a bunch of Javadoc errors. >> It works fine on Java 7. Is this expected? >> >> Other verifications I performed: >> >>- Verified all signatures in RC1 >>- Verified all checksums in RC1 >>- Visually inspected contents of src tarball >>- Built from source on Mac OSX 10.14.6 and RHEL7 (Java 8) >>- mvn -DskipTests package >>- Visually inspected contents of binary tarball >> >> Thanks, >> Erik >> >> -------------- >> *From:* Konstantin Shvachko >> *Sent:* Wednesday, October 23, 2019 6:10 PM >> *To:* Jonathan Hung >> *Cc:* Hdfs-dev ; mapreduce-dev < >> mapreduce-...@hadoop.apache.org>; yarn-dev ; >> Hadoop Common >> *Subject:* Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1) >> >> +1 on RC1 >> >> - Verified signatures >> - Verified maven artifacts on Nexus for sources >> - Checked rat reports >> - Checked documentation >> - Checked packaging contents >> - Built from sources on RHEL 7 box >> - Ran unit tests for new HDFS features with Java 8 >> >> Thanks, >> --Konstantin >> >> On Tue, Oct 22, 2019 at 2:55 PM Jonathan Hung >> wrote: >> >> > Hi folks, >> > >> > This is the second release candidate for the first release of Apache >> Hadoop >> > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes >> > features such as: >> > >> > - User-defined resource types >> > - Native GPU support as a schedulable resource type >> > - Consistent reads from standby node >> > - Namenode port based selective encryption >> > - Improvements related to rolling upgrade support from 2.x to 3.x >> > - Cost based fair call queue >> > >> > The RC1 artifacts are at: >> https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fhome.apache.org%2F~jhung%2Fhadoop-2.10.0-RC1%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=ZX7lF4N3fV38ggkplLU56ybhKBZrx%2FUKMkfxm2WJ7eU%3D&reserved=0 >> > >> > RC tag is release-2.10.0-RC1. >> > >> > The maven artifacts are hosted here: >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1243%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Thanks for looking Erik. For the release notes, yeah I think it's because there's no release notes for the corresponding JIRAs. I've added details for these features to the index.md.vm file which should show up on the homepage for 2.10.0 (e.g. https://hadoop.apache.org/docs/r2.9.0/index.html). We could add release notes for these JIRAs, but that would require recreating the tar.gzs since the release notes are bundled in there. For the javadoc issue, I was able to repro this issue, seems it's because the org.apache.hadoop.yarn.client.ClientRMProxy import was removed in FederationProxyProviderUtil in YARN-7900 in branch-2 (but not in other branches). But it's referenced in javadocs in this file so it throws this error. Re-adding this import and building with java 8 allows it to succeed. I checked javadoc html for FederationProxyProviderUtil in the produced artifacts and it appears to be correct. I think we could easily overwrite the current RC1 artifacts with ones containing proper release notes. Not sure what to do about the javadoc issue though, that would require overwriting the release-2.10.0-RC1 tag which I don't want to do. What do others think? Jonathan Hung On Fri, Oct 25, 2019 at 9:21 AM Erik Krogen wrote: > Thanks for putting this together, Jonathan! > > I noticed that the RELEASENOTES.md makes no mention of any of the major > features you mentioned in your email about the RC. Is this expected? I > guess it is caused by the lack of a release note on the JIRAs for those > features. > > I also noticed that building a distribution package (mvn -DskipTests > package -Pdist) fails on Java 8 (1.8.0_172) with a bunch of Javadoc errors. > It works fine on Java 7. Is this expected? > > Other verifications I performed: > >- Verified all signatures in RC1 >- Verified all checksums in RC1 >- Visually inspected contents of src tarball >- Built from source on Mac OSX 10.14.6 and RHEL7 (Java 8) >- mvn -DskipTests package >- Visually inspected contents of binary tarball > > Thanks, > Erik > > -- > *From:* Konstantin Shvachko > *Sent:* Wednesday, October 23, 2019 6:10 PM > *To:* Jonathan Hung > *Cc:* Hdfs-dev ; mapreduce-dev < > mapreduce-...@hadoop.apache.org>; yarn-dev ; > Hadoop Common > *Subject:* Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1) > > +1 on RC1 > > - Verified signatures > - Verified maven artifacts on Nexus for sources > - Checked rat reports > - Checked documentation > - Checked packaging contents > - Built from sources on RHEL 7 box > - Ran unit tests for new HDFS features with Java 8 > > Thanks, > --Konstantin > > On Tue, Oct 22, 2019 at 2:55 PM Jonathan Hung > wrote: > > > Hi folks, > > > > This is the second release candidate for the first release of Apache > Hadoop > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > > features such as: > > > > - User-defined resource types > > - Native GPU support as a schedulable resource type > > - Consistent reads from standby node > > - Namenode port based selective encryption > > - Improvements related to rolling upgrade support from 2.x to 3.x > > - Cost based fair call queue > > > > The RC1 artifacts are at: > https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fhome.apache.org%2F~jhung%2Fhadoop-2.10.0-RC1%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=ZX7lF4N3fV38ggkplLU56ybhKBZrx%2FUKMkfxm2WJ7eU%3D&reserved=0 > > > > RC tag is release-2.10.0-RC1. > > > > The maven artifacts are hosted here: > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1243%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=DsJDfoj8eg3E%2F%2BNEwOAI41LhcRJ2hOWycS923ds3Seg%3D&reserved=0 > > > > My public key is available here: > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Frelease%2Fhadoop%2Fcommon%2FKEYS&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=1694z6xhj5NtxwYBpwnRBx%2BgK0npGIUs5O580K3KPJw%3D&reserved=0 > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > PDT. > > > > Thanks, > > Jonathan Hung > > > > [1] > > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.a
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Thanks for putting this together, Jonathan! I noticed that the RELEASENOTES.md makes no mention of any of the major features you mentioned in your email about the RC. Is this expected? I guess it is caused by the lack of a release note on the JIRAs for those features. I also noticed that building a distribution package (mvn -DskipTests package -Pdist) fails on Java 8 (1.8.0_172) with a bunch of Javadoc errors. It works fine on Java 7. Is this expected? Other verifications I performed: * Verified all signatures in RC1 * Verified all checksums in RC1 * Visually inspected contents of src tarball * Built from source on Mac OSX 10.14.6 and RHEL7 (Java 8) * mvn -DskipTests package * Visually inspected contents of binary tarball Thanks, Erik From: Konstantin Shvachko Sent: Wednesday, October 23, 2019 6:10 PM To: Jonathan Hung Cc: Hdfs-dev ; mapreduce-dev ; yarn-dev ; Hadoop Common Subject: Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1) +1 on RC1 - Verified signatures - Verified maven artifacts on Nexus for sources - Checked rat reports - Checked documentation - Checked packaging contents - Built from sources on RHEL 7 box - Ran unit tests for new HDFS features with Java 8 Thanks, --Konstantin On Tue, Oct 22, 2019 at 2:55 PM Jonathan Hung wrote: > Hi folks, > > This is the second release candidate for the first release of Apache Hadoop > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: > https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fhome.apache.org%2F~jhung%2Fhadoop-2.10.0-RC1%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=ZX7lF4N3fV38ggkplLU56ybhKBZrx%2FUKMkfxm2WJ7eU%3D&reserved=0 > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1243%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=DsJDfoj8eg3E%2F%2BNEwOAI41LhcRJ2hOWycS923ds3Seg%3D&reserved=0 > > My public key is available here: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Frelease%2Fhadoop%2Fcommon%2FKEYS&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C637074762694349124&sdata=1694z6xhj5NtxwYBpwnRBx%2BgK0npGIUs5O580K3KPJw%3D&reserved=0 > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung > > [1] > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fissues%2F%3Fjql%3Dproject%2520in%2520(HDFS%252C%2520YARN%252C%2520HADOOP%252C%2520MAPREDUCE)%2520AND%2520resolution%2520%253D%2520Fixed%2520AND%2520fixVersion%2520%253D%25202.10.0%2520AND%2520fixVersion%2520not%2520in%2520(2.9.2%252C%25202.9.1%252C%25202.9.0&data=02%7C01%7Cekrogen%40linkedin.com%7C1fee1e5911d8415a418b08d7581f0c7e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637074762694349124&sdata=U2Pt0Vx3k3XWJGoH0S8iFgXbrU0X8TF3ACcAAg9axog%3D&reserved=0) >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 (non-binding) on RC1 I built from source on Mac and RHEL7, ran hdfs, nodemanager, and resourcemanager unit tests, and set up a one-node cluster and ran some test jobs (pi and sleep). - Jim On Tue, Oct 22, 2019 at 4:55 PM Jonathan Hung wrote: > Hi folks, > > This is the second release candidate for the first release of Apache Hadoop > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > My public key is available here: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung > > [1] > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0) >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
+1 on RC1 - Verified signatures - Verified maven artifacts on Nexus for sources - Checked rat reports - Checked documentation - Checked packaging contents - Built from sources on RHEL 7 box - Ran unit tests for new HDFS features with Java 8 Thanks, --Konstantin On Tue, Oct 22, 2019 at 2:55 PM Jonathan Hung wrote: > Hi folks, > > This is the second release candidate for the first release of Apache Hadoop > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > My public key is available here: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung > > [1] > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0) >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Hi Eric, thanks for trying it out. We talked about this in today's YARN community sync up, summarizing here for everyone else: I don't think it's worth delaying the 2.10.0 release further, we can address this in a subsequent 2.10.x release. Wangda mentioned it might be related to changes in dominant resource calculator, but root cause remains to be seen. Jonathan Hung On Wed, Oct 23, 2019 at 9:02 AM epa...@apache.org wrote: > Hi Jonathan, > > Thanks very much for all of your work on this release. > > I have a concern about cross-queue (inter-queue) preemption in 2.10. > > In 2.8, on a 6 node pseudo-cluster, preempting from one queue to meet the > needs of another queue seems to work as expected. However, 2.10 in the same > pseudo-cluster (with the same config properties), only one container was > preempted for the AM and then nothing else. > > I don't know how the community feels about holding up the 2.10.0 release > for this issue, but we need to get to the bottom of this before we can go > to 2.10.x. I am still investigating. > > Thanks, > -Eric > > > > > On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung < > jyhung2...@gmail.com> wrote: > > Hi folks, > > > > This is the second release candidate for the first release of Apache > Hadoop > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > > features such as: > > > > - User-defined resource types > > - Native GPU support as a schedulable resource type > > - Consistent reads from standby node > > - Namenode port based selective encryption > > - Improvements related to rolling upgrade support from 2.x to 3.x > > - Cost based fair call queue > > > > The RC1 artifacts are at: > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > > > RC tag is release-2.10.0-RC1. > > > > The maven artifacts are hosted here: > > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > > > My public key is available here: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm > PDT. > > > > Thanks, > > Jonathan Hung >
Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)
Hi Jonathan, Thanks very much for all of your work on this release. I have a concern about cross-queue (inter-queue) preemption in 2.10. In 2.8, on a 6 node pseudo-cluster, preempting from one queue to meet the needs of another queue seems to work as expected. However, 2.10 in the same pseudo-cluster (with the same config properties), only one container was preempted for the AM and then nothing else. I don't know how the community feels about holding up the 2.10.0 release for this issue, but we need to get to the bottom of this before we can go to 2.10.x. I am still investigating. Thanks, -Eric On Tuesday, October 22, 2019, 4:55:29 PM CDT, Jonathan Hung wrote: > Hi folks, > > This is the second release candidate for the first release of Apache Hadoop > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It includes > features such as: > > - User-defined resource types > - Native GPU support as a schedulable resource type > - Consistent reads from standby node > - Namenode port based selective encryption > - Improvements related to rolling upgrade support from 2.x to 3.x > - Cost based fair call queue > > The RC1 artifacts are at: http://home.apache.org/~jhung/hadoop-2.10.0-RC1/ > > RC tag is release-2.10.0-RC1. > > The maven artifacts are hosted here: > https://repository.apache.org/content/repositories/orgapachehadoop-1243/ > > My public key is available here: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm PDT. > > Thanks, > Jonathan Hung - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
