Re: 2017-09-07 Hadoop 3 release status update

[Steve Loughran]

On 8 Sep 2017, at 00:50, Andrew Wang 
> wrote:

  - HADOOP-14738  (Remove
  S3N and obsolete bits of S3A; rework docs): Steve has been actively revving
  this with our new committer Aaron Fabbri ready to review. The scope has
  expanded from HADOOP-14826, so it's not just a doc update.

For people not tracking this, it's merged with other cleanup code so pulls the 
entirety of the s3n:// connector and the original 
S3AOutputStreamessentially the unmaintained and obsolete bits of code. The 
ones where any bugrep would be dealt with "have you switched to..."

2017-09-07 Hadoop 3 release status update

[Andrew Wang]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-07

Slightly early update since I'll be out tomorrow. We're one week out, and
focus is on blocker burndown.

Highlights:

   - 3.1.0 release planning is underway, led by Wangda. Target release date
   is in January.

Red flags:

   - YARN native services merge vote got a -1 for beta1, I recommended we
   drop it from beta1 and retarget for a later release.
   - 11 blockers on the dashboard, one more than last week [image: (sad)]

Previously tracked beta1 blockers that have been resolved or dropped:

   - HADOOP-14826 was duped to HADOOP-14738.
   - YARN-5536  (Multiple
   format support (JSON, etc.) for exclude node file in NM graceful
   decommission with timeout): Downgraded in priority in favor of YARN-7162
   which Robert has posted a patch for.
   - MAPREDUCE-6941 (The default setting doesn't work for MapReduce job): I
   resolved this and Junping confirmed this is fine.


beta1 blockers:

   - HADOOP-14738  (Remove
   S3N and obsolete bits of S3A; rework docs): Steve has been actively revving
   this with our new committer Aaron Fabbri ready to review. The scope has
   expanded from HADOOP-14826, so it's not just a doc update.
   - HADOOP-14284  (Shade
   Guava everywhere): No change since last week. This is an umbrella JIRA.
   - HADOOP-14771 
(hadoop-client
   does not include hadoop-yarn-client): Patch up, needs review, still waiting
   on Busbey. Bharat gave it a review.
   - YARN-7162  (Remove
   XML excludes file format): Robert has posted a patch and is waiting for a
   review.
   - HADOOP-14238 
(Rechecking
   Guava's object is not exposed to user-facing API): Bharat took this up and
   turned it into an umbrella.
  - HADOOP-14847
 (Remove
  Guava Supplier and change to java Supplier in AMRMClient and
  AMRMClientAysnc) Bharat posted a patch on a subtask to fix the
known Guava
  Supplier issue in AMRMClient. Needs a review.
   - HADOOP-14835  (mvn
   site build throws SAX errors): I'm working on this. Debugged it and have a
   proposed patch up, discussing with Allen.
   - HDFS-12218  (Rename
   split EC / replicated block metrics in BlockManager): I'm working on this,
   just need to commit it, already have a +1 from Eddy.


beta1 features:

   - Erasure coding
  - There are three must-dos, all being actively worked on.
  - HDFS-7859 is being actively reviewed and revved by Sammi and Kai
  and Eddy.
  - HDFS-12395 was split out of HDFS-7859 to do the edit log changes.
  - HDFS-12218 is discussed above.
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Ray and Allen reviewed Sean's HDFS rolling upgrade scripts.
  - Sean did a run through of the HDFS JACC report and it looked fine.
   - Classpath isolation (HADOOP-11656)
  - Sean has retriaged the subtasks and has been posting patches.
   - Compat guide (HADOOP-13714
   )
  - Daniel has been collecting feedback on dev lists, but still needs a
  detailed review of the patch.
   - YARN native services
  - Jian sent out the merge vote, but it's been -1'd for beta1 by
  Allen. I propose we drop this from beta1 scope and retarget.
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
  - Merge vote was sent out. Since branch-3.0 has been cut, this can be
  merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
  run in production at MSFT.
  - If it passes Cloudera internal integration testing, I'm fine
  putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
  - Jonathan mentioned that his main goal is to get this in for 2.9.0,
  which seems likely to go out after 3.0.0 GA since there hasn't been any
  serious release planning yet. Jonathan said that delaying this
until 3.1.0
  is fine.