Re: Reviving HADOOP-7435: Making Jenkins pre-commit build work with branches

[Sean Busbey]

+1

If we can make things look like HBase support for precommit testing on
branches (HBASE-12944), that would make it easier for new and occasional
contributors who might end up working in other ecosystem projects. AFAICT,
Jonathan's proposal for branch names in patch names does this.



On Wed, Mar 4, 2015 at 3:41 PM, Karthik Kambatla ka...@cloudera.com wrote:

 Thanks for reviving this on email, Vinod. Newer folks like me might not be
 aware of this JIRA/effort.

 This would be wonderful to have so (1) we know the status of release
 branches (branch-2, etc.) and also (2) feature branches (YARN-2928).
 Jonathan's or Matt's proposal for including branch name looks reasonable to
 me.

 If none has any objections, I think we can continue on JIRA and get this
 in.

 On Wed, Mar 4, 2015 at 1:20 PM, Vinod Kumar Vavilapalli 
 vino...@hortonworks.com wrote:

  Hi all,
 
  I'd like us to revive the effort at
  https://issues.apache.org/jira/browse/HADOOP-7435 to make precommit
  builds being able to work with branches. Having the Jenkins verify
 patches
  on branches is very useful even if there may be relaxed review oversight
 on
  the said-branch.
 
  Unless there are objections, I'd request help from Giri who already has a
  patch sitting there for more than a year before. This may need us to
  collectively agree on some convention - the last comment says that the
  branch patch name should be in some format for this to work.
 
  Thanks,
  +Vinod
 



 --
 Karthik Kambatla
 Software Engineer, Cloudera Inc.
 
 http://five.sentenc.es




-- 
Sean

Re: committing HADOOP-11746 test-patch improvements

[Sean Busbey]

On Wed, Apr 22, 2015 at 2:10 AM, Allen Wittenauer a...@altiscale.com wrote:



 * There have been a few runs which seems to indicate that *something* is
 destroying the artifact directory in the middle of  runs…. which is very
 very odd and something I hadn’t seen in any of my testing.  In any case, I
 clearly need to add some safety code here to report back that something
 went awry and report back which node, console, etc this happened on.
 Someone more familiar with the Jenkins setup might be able to shed some
 light on why that might happen. All of these runs appear to be on H3, so
 might be related? Impacted issues with this have been:

 - HDFS-8200 (https://builds.apache.org/job/PreCommit-HDFS-Build/10335/)
 - HDFS-8147 (https://builds.apache.org/job/PreCommit-HDFS-Build/10338/)
 - YARN-3301 (https://builds.apache.org/job/PreCommit-YARN-Build/7441/)


From the HDFS precommit build:


 PATCHPROCESS=${WORKSPACE}/../patchprocess
 mkdir -p ${PATCHPROCESS}


Working on directories outside of the workspace for the job is not good,
though I'm not sure if that's the source of the issue. Do I need to
coordinate fixing this with anyone?

-- 
Sean

Re: Set minimum version of Hadoop 3 to JDK 8

[Sean Busbey]

A few options:

* Only change the builds for master to use jdk8
* build with both jdk7 and jdk8 by copying jobs
* build with both jdk7 and jdk8 using a jenkins matrix build

Robert, if you'd like help with any of these please send me a ping off-list.

On Tue, Apr 21, 2015 at 8:19 PM, Vinod Kumar Vavilapalli 
vino...@hortonworks.com wrote:

 We don't want JDK 8 only code going into branch-2 line. Moving Jenkins to
 1.8 right-away will shield such code, how do we address that?

 Thanks,
 +Vinod

 On Apr 21, 2015, at 5:54 PM, Robert Kanter rkan...@cloudera.com wrote:

  Sure, I'll try to change the Jenkins builds to 1.8 first.
 
  On Tue, Apr 21, 2015 at 3:31 PM, Andrew Wang andrew.w...@cloudera.com
  wrote:
 
  Hey Robert,
 
  As a first step, could we try switching all our precommit and nightly
  builds over to use 1.8? This is a prerequisite for HADOOP-11858, and
 safe
  to do in any case since it'll still target 1.7.
 
  I'll note that HADOOP-10530 details the pain Steve went through
 switching
  us to JDK7. Might be some lessons learned about how to do this
 transition
  more smoothly.
 
  Thanks,
  Andrew
 
  On Tue, Apr 21, 2015 at 3:15 PM, Robert Kanter rkan...@cloudera.com
  wrote:
 
  + yarn-dev, hdfs-dev, mapred-dev
 
  On Tue, Apr 21, 2015 at 3:14 PM, Robert Kanter rkan...@cloudera.com
  wrote:
 
  Hi all,
 
  Moving forward on some of the discussions on Hadoop 3, I've created
  HADOOP-11858 to set the minimum version of Hadoop 3 to JDK 8.  I just
  wanted to let everyone know in case there's some reason we shouldn't
 go
  ahead with this.
 
  thanks
  - Robert
 
 
 




-- 
Sean

Re: F 6/19: Jenkins clogged up

[Sean Busbey]

Thanks for hte heads up.

On Fri, Jun 19, 2015 at 1:43 PM, Chris Nauroth cnaur...@hortonworks.com
wrote:

 Hi everyone,

 I was just in contact with Apache infrastructure.  Jenkins wasn't running
 jobs for a while, so there is a large backlog in the queue now (over 200
 jobs).  Infra has fixed the problems, so jobs are running now, but our
 test-patch runs might have to sit in the queue a long time today.

 --Chris Nauroth




-- 
Sean

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

Any update on a release plan for 2.6.1?

On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula 
brahmareddy.batt...@huawei.com wrote:

 HI vinod

 any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1
 as stable give..?


 Thanks  Regards
  Brahma Reddy Battula

 
 From: Zhihai Xu [z...@cloudera.com]
 Sent: Wednesday, May 13, 2015 12:04 PM
 To: mapreduce-dev@hadoop.apache.org
 Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
 hdfs-...@hadoop.apache.org
 Subject: Re: Planning Hadoop 2.6.1 release

 Hi Akira,

 Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore
 bug.
 It will work better with YARN-2992.

 thanks
 zhihai


 On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA 
 ajisa...@oss.nttdata.co.jp
 wrote:

  Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like
  to include the following:
 
  * HADOOP-11343. Overflow is not properly handled in calculating final iv
  for AES CTR
  * YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to
  execute any further apps
  * YARN-2992. ZKRMStateStore crashes due to session expiry
  * YARN-3013. AMRMClientImpl does not update AMRM token properly
  * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to
  die
  * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
 to
  a reducer
 
  All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0.
 
  Regards,
  Akira
 
 
  On 5/4/15 11:15, Brahma Reddy Battula wrote:
 
  Hello Vinod,
 
  I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs
  will not be usable when cluster installed in HA mode and try to get
  filecontext like below..
 
 
  Path path = new
 
 Path(har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har);
  FileSystem fs = path.getFileSystem(new Configuration());
  path = fs.makeQualified(path);
  FileContext fc = FileContext.getFileContext(path.toUri(),new
  Configuration());
 
 
 
  Thanks  Regards
  Brahma Reddy Battula
  
  From: Chris Nauroth [cnaur...@hortonworks.com]
  Sent: Friday, May 01, 2015 4:32 AM
  To: mapreduce-dev@hadoop.apache.org; common-...@hadoop.apache.org;
  yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
  Subject: Re: Planning Hadoop 2.6.1 release
 
  Thank you, Arpit.  In addition, I suggest we include the following:
 
  HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
  pipe is full
  HADOOP-11604. Prevent ConcurrentModificationException while closing
 domain
  sockets during shutdown of DomainSocketWatcher thread.
  HADOOP-11648. Set DomainSocketWatcher thread name explicitly
  HADOOP-11802. DomainSocketWatcher thread terminates sometimes after
 there
  is an I/O error during requestShortCircuitShm
 
  HADOOP-11604 and 11648 are not critical by themselves, but they are
  pre-requisites to getting a clean cherry-pick of 11802, which we believe
  finally fixes the root cause of this issue.
 
 
  --Chris Nauroth
 
 
 
 
  On 4/30/15, 3:55 PM, Arpit Agarwal aagar...@hortonworks.com wrote:
 
   HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
  requested in [1].
 
  HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
  should be non static
  HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
  synchronization
 
  HDFS-7009. Active NN and standby NN have different live nodes.
  HDFS-7035. Make adding a new data directory to the DataNode an atomic
 and
  improve error handling
  HDFS-7425. NameNode block deletion logging uses incorrect appender.
  HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
  block files are present in the same volume.
  HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang
 datanodes
  HDFS-7503. Namenode restart after large deletions can cause slow
  processReport.
  HDFS-7575. Upgrade should generate a unique storage ID for each volume.
  HDFS-7579. Improve log reporting during block report rpc failure.
  HDFS-7587. Edit log corruption can happen if append fails with a quota
  violation.
  HDFS-7596. NameNode should prune dead storages from storageMap.
  HDFS-7611. deleteSnapshot and delete of a file can leave orphaned
 blocks
  in the blocksMap on NameNode restart.
  HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
  DataNode to register successfully with only one NameNode.
  HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
  failure.
  HDFS-7831. Fix the starting index and end condition of the loop in
  FileDiffList.findEarlierSnapshotBlocks().
  HDFS-7885. Datanode should not trust the generation stamp provided by
  client.
  HDFS-7960. The full block report should prune zombie storages even if
  they're not empty.
  HDFS-8072. Reserved RBW space is not released if client terminates
 while
  writing block.
  HDFS-8127. NameNode 

Re: Planning Hadoop 2.6.1 release

[Sean Busbey]

If we haven't frozen yet, HDFS-8850 is a straight forward fix that is
currently only in 2.8+ and would benefit 2.6 and 2.7.

On Wed, Aug 5, 2015 at 2:56 PM, Junping Du j...@hortonworks.com wrote:

 I would like to nominate YARN-3832 as 2.6.1 candidate which is critical
 and I also saw it happened recently on a 2.6.0 cluster. Hope this is not
 too late.

 Thanks,

 Junping
 
 From: Rich Haase rha...@pandora.com
 Sent: Thursday, August 06, 2015 1:52 AM
 To: mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
 Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org
 Subject: Re: Planning Hadoop 2.6.1 release

 +1 to add those fixes.


 Rich Haase | Sr. Software Engineer | Pandora
 m 303.887.1146 | rha...@pandora.com




 On 8/5/15, 11:42 AM, Wangda Tan wheele...@gmail.com wrote:

 Can we add following two fixes to 2.6.1?
 
 https://issues.apache.org/jira/browse/YARN-2922 and
 https://issues.apache.org/jira/browse/YARN-3487.
 
 They're not fatal issue, but they can cause lots of issue in a large
 cluster.
 
 Thanks,
 Wangda
 
 
 On Mon, Aug 3, 2015 at 1:21 PM, Sangjin Lee sj...@apache.org wrote:
 
  See my later update in the thread. HDFS-7704 is in the list.
 
  Thanks,
  Sangjin
 
  On Mon, Aug 3, 2015 at 1:19 PM, Vinod Kumar Vavilapalli 
  vino...@hortonworks.com wrote:
 
   Makes sense, it was caused by HDFS-7704 which got into 2.7.0 only and
 is
   not part of the candidate list. Removed HDFS-7916 from the list.
  
   Thanks
   +Vinod
  
On Jul 24, 2015, at 6:32 PM, Sangjin Lee sj...@apache.org wrote:
   
Out of the JIRAs we proposed, please remove HDFS-7916. I don't
 think it
applies to 2.6.
   
Thanks,
Sangjin
  
  
 




-- 
Sean

Re: IMPORTANT: automatic changelog creation

[Sean Busbey]

On Jul 8, 2015 2:13 AM, Tsuyoshi Ozawa oz...@apache.org wrote:

 +1, thanks Allen and Andrew for taking lots effort!

  Is there any possibility that, we can restrict someone from editing the
  issue in jira once its marked as closed after release?

 Vinay's comment looks considerable for us to me. What do you think?


Mistakes happen, even during the release process.

Presuming the set of folks who can edit closed tickets is already
restricted to contributors, why not assume any edits are the community
making things more accurate?

-- 
Sean

Re: continuing releases on Apache Hadoop 2.6.x

[Sean Busbey]

Early december would be great, presuming the RC process doesn't take too
long. By then it'll already have over a month since the 2.6.2 release and
I'm sure the folks contributing the 18 patches we already have in would
like to see their work out there.

On Fri, Nov 20, 2015 at 7:51 AM, Junping Du  wrote:

> +1. Early Dec sounds too early for 2.6.3 release given we only have 18
> patches since recently release 2.6.2.
> We should nominate more fixes and wait a while for the feedback on 2.6.2.
>
> Thanks,
>
> Junping
> 
> From: Vinod Vavilapalli 
> Sent: Thursday, November 19, 2015 11:34 PM
> To: yarn-...@hadoop.apache.org
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: Re: continuing releases on Apache Hadoop 2.6.x
>
> I see 18 JIRAs across the sub-projects as of now in 2.6.3. Seems like we
> will have a reasonable number of fixes if we start an RC early december.
>
> In the mean while, we should also review 2.7.3 and 2.8.0 blocker /
> critical list and see if it makes sense to backport any of those into 2.6.3.
>
> +Vinod
>
>
> On Nov 17, 2015, at 5:10 PM, Sangjin Lee > wrote:
>
> I'd like to pick up this email discussion again. It is time that we started
> thinking about the next release in the 2.6.x line. IMO we want to walk the
> balance between maintaining a reasonable release cadence and getting a good
> amount of high-quality fixes. The timeframe is a little tricky as the
> holidays are approaching. If we have enough fixes accumulated in
> branch-2.6, some time early December might be a good target for cutting the
> first release candidate. Once we miss that window, I think we are looking
> at next January. I'd like to hear your thoughts on this.
>
>


-- 
Sean

Re: 2.7.3 release plan

[Sean Busbey]

As of 2 days ago, there were already 135 jiras associated with 2.7.3,
if *any* of them end up introducing a regression the inclusion of
HDFS-8791 means that folks will have cluster downtime in order to back
things out. If that happens to any substantial number of downstream
folks, or any particularly vocal downstream folks, then it is very
likely we'll lose the remaining trust of operators for rolling out
maintenance releases. That's a pretty steep cost.

Please do not include HDFS-8791 in any 2.6.z release. Folks having to
be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
unreasonable burden.

I agree that this fix is important, I just think we should either cut
a version of 2.8 that includes it or find a way to do it that gives an
operational path for rolling downgrade.

On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <j...@hortonworks.com> wrote:
> Thanks for bringing up this topic, Sean.
> When I released our latest Hadoop release 2.6.4, the patch of HDFS-8791 
> haven't been committed in so that's why we didn't discuss this earlier.
> I remember in JIRA discussion, we treated this layout change as a Blocker bug 
> that fixing a significant performance regression before but not a normal 
> performance improvement. And I believe HDFS community already did their best 
> with careful and patient to deliver the fix and other related patches (like 
> upgrade fix in HDFS-8578). Take an example of HDFS-8578, you can see 30+ 
> rounds patch review back and forth by senior committers, not to mention the 
> outstanding performance test data in HDFS-8791.
> I would trust our HDFS committers' judgement to land HDFS-8791 on 2.7.3. 
> However, that needs Vinod's final confirmation who serves as RM for 
> branch-2.7. In addition, I didn't see any blocker issue to bring it into 
> 2.6.5 now.
> Just my 2 cents.
>
> Thanks,
>
> Junping
>
> 
> From: Sean Busbey <bus...@cloudera.com>
> Sent: Thursday, March 31, 2016 2:57 PM
> To: hdfs-...@hadoop.apache.org
> Cc: Hadoop Common; yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Re: 2.7.3 release plan
>
> A layout change in a maintenance release sounds very risky. I saw some
> discussion on the JIRA about those risks, but the consensus seemed to
> be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
> we did RMs per release rather than per branch? No one claiming to be a
> release manager ever spoke up AFAICT.
>
> Should this change be included? Should it go into a special 2.8
> release as mentioned in the ticket?
>
> On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
> <ajisa...@oss.nttdata.co.jp> wrote:
>> Thank you Vinod!
>>
>> FYI: 2.7.3 will be a bit special release.
>>
>> HDFS-8791 bumped up the datanode layout version,
>> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> is impossible. We can rollback instead.
>>
>> https://issues.apache.org/jira/browse/HDFS-8791
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>>
>> Regards,
>> Akira
>>
>>
>> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>>
>>> Hi all,
>>>
>>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>>> did go out mid February). Got a little busy since.
>>>
>>> Following up the 2.7.2 maintenance release, we should work towards a
>>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>>> features / improvements.
>>>
>>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>>> / critical issues. Will start moving out any tickets that are not blockers
>>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>>> outstanding as of now.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> [1] 2.7.3 release blockers:
>>> https://issues.apache.org/jira/issues/?filter=12335343
>>>
>>
>
>
>
> --
> busbey



-- 
busbey

Re: 2.7.3 release plan

[Sean Busbey]

A layout change in a maintenance release sounds very risky. I saw some
discussion on the JIRA about those risks, but the consensus seemed to
be "we'll leave it up to the 2.6 and 2.7 release managers." I thought
we did RMs per release rather than per branch? No one claiming to be a
release manager ever spoke up AFAICT.

Should this change be included? Should it go into a special 2.8
release as mentioned in the ticket?

On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
 wrote:
> Thank you Vinod!
>
> FYI: 2.7.3 will be a bit special release.
>
> HDFS-8791 bumped up the datanode layout version,
> so rolling downgrade from 2.7.3 to 2.7.[0-2]
> is impossible. We can rollback instead.
>
> https://issues.apache.org/jira/browse/HDFS-8791
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>
> Regards,
> Akira
>
>
> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>>
>> Hi all,
>>
>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out (which
>> did go out mid February). Got a little busy since.
>>
>> Following up the 2.7.2 maintenance release, we should work towards a
>> 2.7.3. The focus obviously is to have blocker issues [1], bug-fixes and *no*
>> features / improvements.
>>
>> I hope to cut an RC in a week - giving enough time for outstanding blocker
>> / critical issues. Will start moving out any tickets that are not blockers
>> and/or won’t fit the timeline - there are 3 blockers and 15 critical tickets
>> outstanding as of now.
>>
>> Thanks,
>> +Vinod
>>
>> [1] 2.7.3 release blockers:
>> https://issues.apache.org/jira/issues/?filter=12335343
>>
>



-- 
busbey

Re: ASF OS X Build Infrastructure

[Sean Busbey]

Some talk about the MSDN-for-committers program recently passed by on a private
list. It's still active, it just changed homes within Microsoft. The
info should still be in the committer repo. If something is amiss
please let me know and I'll pipe up to the folks already plugged in to
confirming it's active.

On Fri, May 20, 2016 at 12:13 PM, Chris Nauroth
 wrote:
> It's very disappointing to see that vanish.  I'm following up to see if I
> can learn more about what happened or if I can do anything to help
> reinstate it.
>
> --Chris Nauroth
>
>
>
>
> On 5/20/16, 6:11 AM, "Steve Loughran"  wrote:
>
>>
>>> On 20 May 2016, at 10:40, Lars Francke  wrote:
>>>

 Regarding lack of personal access to anything but Linux, I'll take
this as
 an opportunity to remind everyone that ASF committers (not just
limited to
 Hadoop committers) are entitled to a free MSDN license, which can get
you
 a Windows VM for validating Windows issues and any patches that touch
 cross-platform concerns, like the native code.  Contributors who are
not
 committers still might struggle to get access to Windows, but all of us
 reviewing and committing patches do have access.

>>>
>>> Actually, from all I can tell this MSDN offer has been discontinued for
>>> now. All the information has been removed from the committers repo. Do
>>>you
>>> have any more up to date information on this?
>>>
>>
>>
>>That's interesting.
>>
>>I did an SVN update and it went away..looks like something happened on
>>April 26
>>
>>No idea, though the svn log has a bit of detail
>>
>>-
>>To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>>For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch HADOOP-12930

[Sean Busbey]

+1 (non-binding)

reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.

tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.

On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer  wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) 
> to merge the HADOOP-12930 feature branch into trunk. This branch was 
> developed exclusively by me as per the discussion two months ago as a way to 
> make what would be a rather large patch hopefully easier to review.  The vast 
> majority of the branch is code movement in the same file, additional license 
> headers, maven assembly hooks for distribution, and variable renames. Not a 
> whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ 
> commands to allow for subcommands to be added or modified at runtime.  This 
> allows for individual users or entire sites to tweak the execution 
> environment to suit their local needs.  For example, it has been a practice 
> for some locations to change the distcp jar out for a custom one.  Using this 
> functionality, it is possible that the ‘hadoop distcp’ command could run the 
> local version without overwriting the bundled jar and for existing 
> documentation (read: results from Internet searches) to work as written 
> without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop 
> experience
> * operations teams that may be able to leverage existing 
> documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial 
> experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the 
> first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather 
> than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be 
> consistent with other bundled profiles, including the ones introduced in this 
> branch
>
> Documentation, including a ‘hello world’-style example, is in the 
> UnixShellGuide markdown file.  (Of course!)
>
>  I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks.  These are blocked by other 
> issues so that we may add unit testing to the shell code in those respective 
> areas.  I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sean Busbey]

thanks for bringing this up! big +1 on upgrading dependencies for 3.0.

I have an updated patch for HADOOP-11804 ready to post this week. I've
been updating HBase's master branch to try to make use of it, but
could use some other reviews.

On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa  wrote:
> Hi developers,
>
> I'd like to discuss how to make an advance towards dependency
> management in Apache Hadoop trunk code since there has been lots work
> about updating dependencies in parallel. Summarizing recent works and
> activities as follows:
>
> 0) Currently, we have merged minimum update dependencies for making
> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> 1) After that, some people suggest that we should update the other
> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
>
> Main problems we try to solve in the activities above is as follows:
>
> * 1) tries to solve dependency hell between user-level jar and
> system(Hadoop)-level jar.
> * 2) tries to solve updating old libraries.
>
> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> to separate class loader between client-side dependencies and
> server-side dependencies in Hadoop, so we can the change policy of
> updating libraries after doing 2). We can also decide which libraries
> can be shaded after 2).
>
> Hence, IMHO, a straight way we should go to is doing 2 at first.
> After that, we can update both client-side and server-side
> dependencies based on new policy(maybe we should discuss what kind of
> incompatibility is acceptable, and the others are not).
>
> Thoughts?
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Sean Busbey]

> Longer-term, I assume the 2.x line is not ending with 2.8. So we'd still
> have the issue of things committed for 2.9.0 that will be appearing for the
> first time in 3.0.0-alpha1. Assuming a script exists to fix up 2.9 JIRAs,
> it's only incrementally more work to also fix up 2.8 and other unreleased
> versions too.

This. this is why I've been a bit confused on folks not wanting to
invest the time in getting multiple major branches working correctly.
With my "Hadoop community" hat on, I see that we struggle with
maintaining multiple maintenance lines just within 2.y and I worry how
we're going to do once 3.y is going.

With my downstream "HBase community" hat on, I'm pretty sure I still
need there to still be 2.y releases. AFAIK the HBase community hasn't
made any plans to e.g. abandon Hadoop 2.y versions in our next major
release and we'd be very sad if all future features we needed added to
e.g. HDFS forced our users to upgrade across a major version.


On Thu, Jul 21, 2016 at 2:33 PM, Andrew Wang  wrote:
> I really, really want a 3.0.0-alpha1 ASAP, since it's basically impossible
> for downstreams to test incompat changes and new features without a release
> artifact. I've been doing test builds, and branch-3.0.0-alpha1 is ready for
> an RC besides possibly this fix version issue.
>
> I'm not too worried about splitting community bandwidth, for the following
> reasons:
>
> * 3.0.0-alpha1 is very explicitly an alpha, which means no quality or
> compatibility guarantees. It needs less vetting than a 2.x release.
> * Given that 3.0.0 is still in alpha, there aren't many true showstopper
> bugs. Most blockers I see are also apply to both 2.x as well as 3.0.0.
> * Community bandwidth isn't zero-sum. This particularly applies to people
> working on features that are only present in trunk, like EC, shell script
> rewrite, etc.
>
> Longer-term, I assume the 2.x line is not ending with 2.8. So we'd still
> have the issue of things committed for 2.9.0 that will be appearing for the
> first time in 3.0.0-alpha1. Assuming a script exists to fix up 2.9 JIRAs,
> it's only incrementally more work to also fix up 2.8 and other unreleased
> versions too.
>
> Best,
> Andrew
>
> On Thu, Jul 21, 2016 at 11:53 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> The L & N fixes just went out, I’m working to push out 2.7.3 - running
>> into a Nexus issue. Once that goes out, I’ll immediately do a 2.8.0.
>>
>> Like I requested before in one of the 3.x threads, can we just line up
>> 3.0.0-alpha1 right behind 2.8.0?
>>
>> That simplifies most of this confusion, we can avoid splitting the
>> bandwidth from the community on fixing blockers / vetting these concurrent
>> releases. Waiting a little more for 3.0.0 alpha to avoid most of this is
>> worth it, IMO.
>>
>> Thanks
>> +Vinod
>>
>> > On Jul 21, 2016, at 11:34 AM, Andrew Wang 
>> wrote:
>> >
>> > Hi all,
>> >
>> > Since we're planning to spin releases off of both branch-2 and trunk, the
>> > changelog for 3.0.0-alpha1 based on JIRA information isn't accurate. This
>> > is because historically, we've only set 2.x fix versions, and 2.8.0 and
>> > 2.9.0 and etc have not been released. So there's a whole bunch of changes
>> > which will show up for the first time in 3.0.0-alpha1.
>> >
>> > I think I can write a script to (carefully) add 3.0.0-alpha1 to these
>> > JIRAs, but I figured I'd give a heads up here in case anyone felt
>> > differently. I can also update the HowToCommit page to match.
>> >
>> > Thanks,
>> > Andrew
>>
>>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Sean Busbey]

On Thu, Jul 21, 2016 at 4:32 PM, Vinod Kumar Vavilapalli
 wrote:
>> I really, really want a 3.0.0-alpha1 ASAP, since it's basically impossible 
>> for downstreams to test incompat changes and new features without a release 
>> artifact. I've been doing test builds, and branch-3.0.0-alpha1 is ready for 
>> an RC besides possibly this fix version issue.
>
> Not arguing against the need for an alpha release, the question is if it can 
> wait till after 2.8 gets done.
>
> Orthogonally, do we have a report of the incompatible changes? Like the one I 
> generated for some of the branch-2 releases using late jdiff work from Li Lu 
> etc. We should do this and fix any inadvertant incompatibilities. Without 
> seeing this list of incompatibilities, why even make an alpha release and 
> force downstream components to discover issues what can be identified through 
> running reports.
>

I can come up with this, atleast for Source / Binary API compatibility,
provided folks don't mind if I use the Java API Compliance Checker[1]
instead of jdiff.

I'm already familiar with quickly using it, esp with Audience
Annotations from my work in HBase.

Do you want this check from some particular branch-2 release? It
matters since the releases along branch-2 have themselves had some
noise[2].

[1]: https://github.com/lvc/japi-compliance-checker
[2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/

-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Sean Busbey]

Yes, the Java API Compliance Checker allows specifying Annotations to
pare down where incompatible changes happen. It was added some time
ago based on feedback from the Apache HBase project.

The limitations I've found are: 1) at least earlier versions only
supported annotations at the class level (which prevents things like
VisibleForTesting), 2) sometimes it will include more restricted
classes if they are used in less restrictive APIs (e.g. if a IA.Public
class makes use of a IA.Private class in a method signature).

At least when we've used it in HBase, these limitations have been very
easy to spot an explain in a small amount of text. I expect I will be
able to do the same with Hadoop. If we'd like to automate this, the
author has been very responsive to feature requests thus far.


On Mon, Jul 25, 2016 at 3:47 PM, Vinod Kumar Vavilapalli
 wrote:
> Actually, I wouldn’t trust this report as it stands today at all.
>
> I quickly glanced the report, looking for what it highlights as
> incompatible. But the ones that I saw have private / visible for testing
> annotations. Other than acting as useless evidence for those lashing out on
> branch-2, this won’t do much good.
>
> Whenever we start working towards switching to this tool, it should
> incorporate the same exclude-annotations logic that the jdiff code-path does
> today. Do you think that is possible?
>
> Thanks
> +Vinod
>
> On Jul 22, 2016, at 4:53 PM, Vinod Kumar Vavilapalli 
> wrote:
>
> Do you want this check from some particular branch-2 release? It
> matters since the releases along branch-2 have themselves had some
> noise[2].
>
> [1]: https://github.com/lvc/japi-compliance-checker
> 
> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/
> 
>
> --
> busbey
>
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Sean Busbey]

Just so I don't waste time chasing my tail, should I interpret this
email and the associated JIRA as the PMC preferring I not spend
volunteer time providing a compatibility breakdown as previously
discussed?

On Mon, Jul 25, 2016 at 7:54 PM, Wangda Tan <wheele...@gmail.com> wrote:
> I just filed ticket https://issues.apache.org/jira/browse/HADOOP-13423 to
> track running JDIFF on trunk and analyze results for Hadoop-common. I will
> work on that and keep the JIRA and this thread updated. We need to do the
> same work for YARN/MR/HDFS.
>
> On Mon, Jul 25, 2016 at 5:47 PM, Wangda Tan <wheele...@gmail.com> wrote:
>>
>> I agree with what Vinod mentioned: we need to revisit all incompatible
>> changes and revert unnecessary ones. Even if we don't have any compatibility
>> guarantees between 2.x and 3.x. But make user to be less frustrated while
>> trying 3.x is always a better option to me.
>>
>> To achieve this we need to run jdiff for trunk and look at results. I
>> would suggest to do this before 3.0.0-alpha1 release.
>>
>> In addition, for people who will try this 3-alpha1 release artifacts, a
>> guide about migration from 2.x to 3.x will be very helpful, and it can also
>> help for people to better understand what have changed (Just like
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html)
>>
>> Thoughts?
>>
>> Thanks,
>> Wangda
>>
>>
>> On Thu, Jul 21, 2016 at 2:41 PM, Sean Busbey <bus...@cloudera.com> wrote:
>>>
>>> On Thu, Jul 21, 2016 at 4:32 PM, Vinod Kumar Vavilapalli
>>> <vino...@apache.org> wrote:
>>> >> I really, really want a 3.0.0-alpha1 ASAP, since it's basically
>>> >> impossible for downstreams to test incompat changes and new features 
>>> >> without
>>> >> a release artifact. I've been doing test builds, and branch-3.0.0-alpha1 
>>> >> is
>>> >> ready for an RC besides possibly this fix version issue.
>>> >
>>> > Not arguing against the need for an alpha release, the question is if
>>> > it can wait till after 2.8 gets done.
>>> >
>>> > Orthogonally, do we have a report of the incompatible changes? Like the
>>> > one I generated for some of the branch-2 releases using late jdiff work 
>>> > from
>>> > Li Lu etc. We should do this and fix any inadvertant incompatibilities.
>>> > Without seeing this list of incompatibilities, why even make an alpha
>>> > release and force downstream components to discover issues what can be
>>> > identified through running reports.
>>> >
>>>
>>> I can come up with this, atleast for Source / Binary API compatibility,
>>> provided folks don't mind if I use the Java API Compliance Checker[1]
>>> instead of jdiff.
>>>
>>> I'm already familiar with quickly using it, esp with Audience
>>> Annotations from my work in HBase.
>>>
>>> Do you want this check from some particular branch-2 release? It
>>> matters since the releases along branch-2 have themselves had some
>>> noise[2].
>>>
>>> [1]: https://github.com/lvc/japi-compliance-checker
>>> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/
>>>
>>> --
>>> busbey
>>>
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>>
>>
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: HADOOP-13410

[Sean Busbey]

ServiceLoader API stuff won't load out of the unpacked version, right?

On Tue, Aug 9, 2016 at 11:00 AM, Sangjin Lee  wrote:
> I'd like to get feedback from the community (especially those who might
> remember this) on HADOOP-13410:
> https://issues.apache.org/jira/browse/HADOOP-13410
>
> It appears that Hadoop's RunJar adds the original jar to the app's
> classpath even though the unjarred contents of the jar are in the
> classpath. As long as the file is a jar (or its variant), this seems
> completely superfluous. My suspicion is that the line of code that adds the
> jar to the classpath may have been left there by accident.
>
> Could anyone confirm this? Does anyone see an issue with removing the jar
> from the classpath? I've tested the fix with a couple of simple apps, and I
> didn't see a problem.
>
> Thanks,
> Sangjin



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DICUSS] Upgrading Guice to 4.0(HADOOP-12064)

[Sean Busbey]

At the very least, I'm running through an updated shaded hadoop client
this week[1] (HBase is my test application and it wandered onto some
private things that broke in branch-2). And Sangjin has a good lead on
an lower-short-term-cost incremental improvement for runtime isolation
of apps built on yarn/mapreduce[2]. He's been patiently waiting for
more review feedback.


[1]: https://issues.apache.org/jira/browse/HADOOP-11804
[2]: https://issues.apache.org/jira/browse/HADOOP-13070

On Wed, Jun 29, 2016 at 12:33 PM, Vinod Kumar Vavilapalli
 wrote:
> My strong expectation is that we’ll have a version of classpath isolation in 
> our first release of 3.x. I’m planning to spending some cycles right away on 
> this.
>
> Assuming classpath isolation gets in, it is reasonable to bump up our 
> dependencies like Jetty / Guice to the latest stable versions.
>
> Thanks
> +Vinod
>
>> On Jun 27, 2016, at 6:01 AM, Tsuyoshi Ozawa  wrote:
>>
>> Hi developers,
>>
>> I will plan to upgrade Google Guice dependency on trunk. The change
>> also includes asm and cglib upgrade.
>> I checked following points:
>>
>> * Both HDFS and YARN UIs work well.
>> * All webIU-related tests pass as described on HADOOP-12064.
>> * Ran mapreduce job, and it works well.
>>
>> https://issues.apache.org/jira/browse/HADOOP-12064
>>
>> Do you have any concern or opinion?  I would like to merge it to trunk
>> on this Friday if you have no objections.
>>
>> Best,
>> - Tsuyoshi
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Sean Busbey]

It's also the key Andrew has in the project's KEYS file:

http://www.apache.org/dist/hadoop/common/KEYS



On Tue, Aug 30, 2016 at 4:12 PM, Andrew Wang  wrote:
> Hi Eric, thanks for trying this out,
>
> I tried this gpg command to get my key, seemed to work:
>
> # gpg --keyserver pgp.mit.edu --recv-keys 7501105C
> gpg: requesting key 7501105C from hkp server pgp.mit.edu
> gpg: /root/.gnupg/trustdb.gpg: trustdb created
> gpg: key 7501105C: public key "Andrew Wang (CODE SIGNING KEY) <
> andrew.w...@cloudera.com>" imported
> gpg: no ultimately trusted keys found
> gpg: Total number processed: 1
> gpg:   imported: 1  (RSA: 1)
>
> Also found via search:
> http://pgp.mit.edu/pks/lookup?search=wang%40apache.org=index
>
>
> On Tue, Aug 30, 2016 at 2:06 PM, Eric Badger  wrote:
>
>> I don't know why my email client keeps getting rid of all of my spacing.
>> Resending the same email so that it is actually legible...
>>
>> All on OSX 10.11.6:
>> - Verified the hashes. However, Andrew, I don't know where to find your
>> public key, so I wasn't able to verify that they were signed by you.
>> - Built from source
>> - Deployed a pseudo-distributed clusterRan a few sample jobs
>> - Poked around the RM UI
>> - Poked around the attached website locally via the tarball
>>
>>
>> I did find one odd thing, though. It could be a misconfiguration on my
>> system, but I've never had this problem before with other releases (though
>> I deal almost exclusively in 2.x and so I imagine things might be
>> different). When I run a sleep job, I do not see any
>> diagnostics/logs/counters printed out by the client. Initially I ran the
>> job like I would on 2.7 and it failed (because I had not set
>> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see
>> anything until I looked at the RM UI. There I was able to see all of the
>> logs for the failed job and diagnose the issue. Then, once I fixed my
>> parameters and ran the job again, I still didn't see any
>> diagnostics/logs/counters.
>>
>>
>> ebadger@foo: env | grep HADOOP
>> HADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1-
>> src/hadoop-dist/target/hadoop-3.0.0-alpha1/
>> HADOOP_CONF_DIR=/Users/ebadger/conf
>> ebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/
>> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep
>> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
>> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1
>> -m 1 -r 1
>> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
>> ebadger@foo:
>>
>>
>> After running the above command, the RM UI showed a successful job, but as
>> you can see, I did not have anything printed onto the command line.
>> Hopefully this is just a misconfiguration on my part, but I figured that I
>> would point it out just in case.
>>
>>
>> Thanks,
>>
>>
>> Eric
>>
>>
>>
>> On Tuesday, August 30, 2016 4:00 PM, Eric Badger
>>  wrote:
>>
>>
>>
>> All on OSX 10.11.6:
>> Verified the hashes. However, Andrew, I don't know where to find your
>> public key, so I wasn't able to verify that they were signed by you.Built
>> from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked
>> around the RM UIPoked around the attached website locally via the tarball
>> I did find one odd thing, though. It could be a misconfiguration on my
>> system, but I've never had this problem before with other releases (though
>> I deal almost exclusively in 2.x and so I imagine things might be
>> different). When I run a sleep job, I do not see any
>> diagnostics/logs/counters printed out by the client. Initially I ran the
>> job like I would on 2.7 and it failed (because I had not set
>> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see
>> anything until I looked at the RM UI. There I was able to see all of the
>> logs for the failed job and diagnose the issue. Then, once I fixed my
>> parameters and ran the job again, I still didn't see any
>> diagnostics/logs/counters.
>> ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/
>> ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/
>> target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo:
>> $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/
>> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep
>> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME"
>> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1
>> -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be
>> incomplete.ebadger@foo:
>> After running the above command, the RM UI showed a successful job, but as
>> you can see, I did not have anything printed onto the command line.
>> Hopefully this is just a misconfiguration on my part, but I figured that I
>> would point it out just in case.
>> Thanks,
>> Eric
>>
>>
>>
>> On 

[NOTICE] breaking precommit checks

[Sean Busbey]

Hi folks!

a host of precommit checks are currently timing out due to an update
to our job configs (the timeout is currently set to 50 minutes).

I'm in the process of giving things more time based on our historic
usage, but if your check fails in the mean time and

1) the total run time is close to 50 minutes

2) the jenkins job when you click through shows status "aborted"

then please resubmit your patch.

-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [NOTICE] breaking precommit checks

[Sean Busbey]

Should be all set now.

On Tue, Nov 8, 2016 at 5:54 PM, Sean Busbey <bus...@cloudera.com> wrote:
> Hi folks!
>
> a host of precommit checks are currently timing out due to an update
> to our job configs (the timeout is currently set to 50 minutes).
>
> I'm in the process of giving things more time based on our historic
> usage, but if your check fails in the mean time and
>
> 1) the total run time is close to 50 minutes
>
> 2) the jenkins job when you click through shows status "aborted"
>
> then please resubmit your patch.
>
> --
> busbey



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Sean Busbey]

disallowing force pushes to trunk was done back in:

* August 2014: INFRA-8195
* February 2016: INFRA-11136

On Mon, Apr 17, 2017 at 11:18 AM, Jason Lowe
 wrote:
> I found at least one commit that was dropped, MAPREDUCE-6673.  I was able to 
> cherry-pick the original commit hash since it was recorded in the commit 
> email.
> This begs the question of why we're allowing force pushes to trunk.  I 
> thought we asked to have that disabled the last time trunk was accidentally 
> clobbered?
> Jason
>
>
> On Monday, April 17, 2017 10:18 AM, Arun Suresh  
> wrote:
>
>
>  Hi
>
> I had the Apr-14 eve version of trunk on my local machine. I've pushed that.
> Don't know if anything was committed over the weekend though.
>
> Cheers
> -Arun
>
> On Mon, Apr 17, 2017 at 7:17 AM, Anu Engineer 
> wrote:
>
>> Hi Allen,
>>
>> https://issues.apache.org/jira/browse/INFRA-13902
>>
>> That happened with ozone branch too. It was an inadvertent force push.
>> Infra has advised us to force push the latest branch if you have it.
>>
>> Thanks
>> Anu
>>
>>
>> On 4/17/17, 7:10 AM, "Allen Wittenauer"  wrote:
>>
>> >Looks like someone reset HEAD back to Mar 31.
>> >
>> >Sent from my iPad
>> >
>> >> On Apr 16, 2017, at 12:08 AM, Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>> >>
>> >> For more details, see https://builds.apache.org/job/
>> hadoop-qbt-trunk-java8-linux-x86/378/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -1 overall
>> >>
>> >>
>> >> The following subsystems voted -1:
>> >>docker
>> >>
>> >>
>> >> Powered by Apache Yetus 0.5.0-SNAPSHOT  http://yetus.apache.org
>> >>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >-
>> >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>
>
>



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Pre-Commit build is failing

[Sean Busbey]

-dev@yetus to bcc, since I think this is a Hadoop issue and not a yetus
issue.

Please review/commit HADOOP-14686 (which I am providing as a
volunteer/contributor on the Hadoop project).

On Tue, Jul 25, 2017 at 7:54 PM, Allen Wittenauer 
wrote:

>
> Again: just grab the .gitignore file from trunk and update it in
> branch-2.7. It hasn't been touched (outside of one patch) in years.  The
> existing jobs should then work.
>
> The rest of this stuff, yes, I know and yes it's intentional.  The
> directory structure was inherited from the original jobs that Nigel set up
> with the old version of test-patch.  Maybe some day I'll fix it.  But
> that's a project for a different day.  In order to fix it, it means taking
> down the patch testing for Hadoop while I work it out.  You'll notice that
> all of the other Yetus jobs for Hadoop have a much different layout.
>
>
>
>
> > On Jul 25, 2017, at 7:24 PM, suraj acharya  wrote:
> >
> > Hi,
> >
> > Seems like the issue was incorrect/unclean checkout.
> > I made a few changes[1] to the directories the checkout happens to  and
> it is now running.
> > Of course, this build[2] will take some time to run, but at the moment,
> it is running maven install.
> >
> > I am not sure who sets up/ manages the jenkins job of HDFS and dont want
> to change that, but I will keep the dummy job around for a couple of days
> in case anyone wants to see.
> > Also, I see that you'll were using the master branch of Yetus. If there
> is no patch present there that is of importance, then I would recommend to
> use the latest stable release version 0.5.0
> >
> > If you have more questions, feel free to ping dev@yetus.
> > Hope this helps.
> >
> > [1]: https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/configure
> > [2]: https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/12/console
> >
> > -Suraj Acharya
> >
> > On Tue, Jul 25, 2017 at 6:57 PM, suraj acharya 
> wrote:
> > For anyone looking. I created another job here. [1].
> > Set it with debug to see the issue.
> > The error is being seen here[2].
> > From the looks of it, it looks like, the way the checkout is happening
> is not very clean.
> > I will continue to look at it, but in case anyone wants to jump in.
> >
> > [1] : https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-Copy/
> > [2] : https://builds.apache.org/job/PreCommit-HDFS-Build-Suraj-
> Copy/11/console
> >
> > -Suraj Acharya
> >
> > On Tue, Jul 25, 2017 at 6:28 PM, Konstantin Shvachko <
> shv.had...@gmail.com> wrote:
> > Hi Yetus developers,
> >
> > We cannot build Hadoop branch-2.7 anymore. Here is a recent example of a
> > failed build:
> > https://builds.apache.org/job/PreCommit-HDFS-Build/20409/console
> >
> > It seems the build is failing because Yetus cannot apply the patch from
> the
> > jira.
> >
> > ERROR: HDFS-11896 does not apply to branch-2.7.
> >
> > As far as I understand this is Yetus problem. Probably in 0.3.0.
> > I can apply this patch successfully, but Yetus test-patch.sh script
> clearly
> > failed to apply. Cannot say why because Yetus does not report it.
> > I also ran Hadoop's test-patch.sh script locally and it passed
> successfully
> > on branch-2.7.
> >
> > Could anybody please take a look and help fixing the build.
> > This would be very helpful for the release (2.7.4) process.
> >
> > Thanks,
> > --Konst
> >
> > On Mon, Jul 24, 2017 at 10:41 PM, Konstantin Shvachko <
> shv.had...@gmail.com>
> > wrote:
> >
> > > Or should we backport the entire HADOOP-11917
> > >  ?
> > >
> > > Thanks,
> > > --Konst
> > >
> > > On Mon, Jul 24, 2017 at 6:56 PM, Konstantin Shvachko <
> shv.had...@gmail.com
> > > > wrote:
> > >
> > >> Allen,
> > >>
> > >> Should we add "patchprocess/" to .gitignore, is that the problem for
> 2.7?
> > >>
> > >> Thanks,
> > >> --Konstantin
> > >>
> > >> On Fri, Jul 21, 2017 at 6:24 PM, Konstantin Shvachko <
> > >> shv.had...@gmail.com> wrote:
> > >>
> > >>> What stuff? Is there a jira?
> > >>> It did work like a week ago. Is it a new Yetus requirement.
> > >>> Anyways I can commit a change to fix the build on our side.
> > >>> Just need to know what is missing.
> > >>>
> > >>> Thanks,
> > >>> --Konst
> > >>>
> > >>> On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer <
> > >>> a...@effectivemachines.com> wrote:
> > >>>
> > 
> >  > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko <
> >  shv.had...@gmail.com> wrote:
> >  >
> >  > + d...@yetus.apache.org
> >  >
> >  > Guys, could you please take a look. Seems like Yetus problem with
> >  > pre-commit build for branch-2.7.
> > 
> > 
> >  branch-2.7 is missing stuff in .gitignore.
> > >>>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional 

[DISCUSS] moving to Apache Yetus Audience Annotations

[Sean Busbey]

When Apache Yetus formed, it started with several key pieces of Hadoop that
looked reusable. In addition to our contribution testing infra, the project
also stood up a version of our audience annotations for delineating the
public facing API[1].

I recently got the Apache HBase community onto the Yetus version of those
annotations rather than their internal fork of the Hadoop ones[2]. It
wasn't pretty, mostly a lot of blind sed followed by spot checking and
reliance on automated tests.

What do folks think about making the jump ourselves? I'd be happy to work
through things, either as one unreviewable monster or per-module
transitions (though a piece-meal approach might complicate our javadoc
situation).


[1]: http://yetus.apache.org/documentation/0.5.0/interface-classification/
[2]: https://issues.apache.org/jira/browse/HBASE-17823

-- 
busbey

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

[Sean Busbey]

I'd refer to it as an incompatible change; we expressly label the
annotations as IA.Public.

If you think it's too late to get in for 3.0, I can make a jira and put it
on the back burner for when trunk goes to 4.0?

On Fri, Sep 22, 2017 at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Is this itself an incompatible change? I imagine the bytecode will be
> different.
>
> I think we're too late to do this for beta1 given that I want to cut an
> RC0 today.
>
> On Fri, Sep 22, 2017 at 7:03 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> When Apache Yetus formed, it started with several key pieces of Hadoop
>> that
>> looked reusable. In addition to our contribution testing infra, the
>> project
>> also stood up a version of our audience annotations for delineating the
>> public facing API[1].
>>
>> I recently got the Apache HBase community onto the Yetus version of those
>> annotations rather than their internal fork of the Hadoop ones[2]. It
>> wasn't pretty, mostly a lot of blind sed followed by spot checking and
>> reliance on automated tests.
>>
>> What do folks think about making the jump ourselves? I'd be happy to work
>> through things, either as one unreviewable monster or per-module
>> transitions (though a piece-meal approach might complicate our javadoc
>> situation).
>>
>>
>> [1]: http://yetus.apache.org/documentation/0.5.0/interface-classi
>> fication/
>> [2]: https://issues.apache.org/jira/browse/HBASE-17823
>>
>> --
>> busbey
>>
>
>


-- 
busbey

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Sean Busbey]

Just curious, Junping what would "solid evidence" look like? Is the
supposition here that the memory leak is within HDFS test code rather than
library runtime code? How would such a distinction be shown?

On Tue, Oct 24, 2017 at 4:06 PM, Junping Du  wrote:

> Allen,
>  Do we have any solid evidence to show the HDFS unit tests going
> through the roof are due to serious memory leak by HDFS? Normally, I don't
> expect memory leak are identified in our UTs - mostly, it (test jvm gone)
> is just because of test or deployment issues.
>  Unless there is concrete evidence, my concern on seriously memory
> leak for HDFS on 2.8 is relatively low given some companies (Yahoo,
> Alibaba, etc.) have deployed 2.8 on large production environment for
> months. Non-serious memory leak (like forgetting to close stream in
> non-critical path, etc.) and other non-critical bugs always happens here
> and there that we have to live with.
>
> Thanks,
>
> Junping
>
> 
> From: Allen Wittenauer 
> Sent: Tuesday, October 24, 2017 8:27 AM
> To: Hadoop Common
> Cc: Hdfs-dev; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
> > On Oct 23, 2017, at 12:50 PM, Allen Wittenauer 
> wrote:
> >
> >
> >
> > With no other information or access to go on, my current hunch is that
> one of the HDFS unit tests is ballooning in memory size.  The easiest way
> to kill a Linux machine is to eat all of the RAM, thanks to overcommit and
> that’s what this “feels” like.
> >
> > Someone should verify if 2.8.2 has the same issues before a release goes
> out …
>
>
> FWIW, I ran 2.8.2 last night and it has the same problems.
>
> Also: the node didn’t die!  Looking through the workspace (so the
> next run will destroy them), two sets of logs stand out:
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-
> linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> and
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-
> linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/
>
> It looks like my hunch is correct:  RAM in the HDFS unit tests are
> going through the roof.  It’s also interesting how MANY log files there
> are.  Is surefire not picking up that jobs are dying?  Maybe not if memory
> is getting tight.
>
> Anyway, at the point, branch-2.8 and higher are probably fubar’d.
> Additionally, I’ve filed YETUS-561 so that Yetus-controlled Docker
> containers can have their RAM limits set in order to prevent more nodes
> going catatonic.
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Sean Busbey]

Here's the email from last night to common-dev@hadoop:

https://s.apache.org/ARe1

On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka  wrote:

> Yes, qbt runs nightly and it sends e-mail to dev lists.
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>
> Regards,
> Akira
>
>
> On 2017/10/19 7:54, Wangda Tan wrote:
>
>> Hi,
>>
>> Do we still have nightly (or even weekly) unit test run for Hadoop
>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>> reports set to dev lists for a while.
>>
>> Thanks,
>> Wangda
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Sean Busbey]

-1 (non-binding)

force pushes are extremely disruptive. there's no way to know who's
updated their local git repo to include these changes since the commit
went in. if a merge commit is so disruptive that we need to subject folks
to the inconvenience of a force push then we should have more tooling
in place to avoid them (like client side git hooks for all committers).

On Thu, Jul 5, 2018 at 4:37 PM, Subru Krishnan  wrote:
> Folks,
>
> There was a merge commit accidentally pushed to trunk, you can find the
> details in the mail thread [1].
>
> I have raised an INFRA ticket [2] to reset/force push to clean up trunk.
>
> Can we have a quick vote for INFRA sign-off to proceed as this is blocking
> all commits?
>
> Thanks,
> Subru
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
> [2] https://issues.apache.org/jira/browse/INFRA-16727



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Make EOL branches to public?

[Sean Busbey]

For what it's worth, in HBase we've been approximating which Hadoop
lines are EOL by looking at release rates and specifically CVE
announcements that include an affected release line but do not include
a fix for that release line. Our current approximation[1] lists 2.6,
2.7, and 3.0 as dead. So that lines up well with your proposed list.


[1]: http://hbase.apache.org/book.html#hadoop

On Fri, Aug 16, 2019 at 5:14 AM Wangda Tan  wrote:
>
> Hi folks,
>
> Want to hear your thoughts about what we should do to make some branches
> EOL. We discussed a couple of times before in dev lists and PMC list.
> However, we couldn't get a formal process of EOL. According to the
> discussion. It is hard to decide it based on time like "After 1st release,
> EOL in 2 years". Because community members still want to maintain it and
> developers still want to get a newer version released.
>
> However, without a public place to figure out which release will be EOL, it
> is very hard for users to choose the right releases to upgrade and develop.
>
> So I want to propose to make an agreement about making a public EOL wiki
> page and create a process to EOL a release:
>
> The process I'm thinking is very simple: If no volunteer to do a
> maintenance release in a short to mid-term (like 3 months to 1 or 1.5
> year). We will claim a release is EOL. After EOL community can still choose
> to do a security-only release.
>
> Here's a list which I can think about:
>
> 1) 2.6.x (Or any release older than 2.6) (Last released at Oct 2016)
> 2) 2.7.x (Last released at Apr 2018)
> 4) 3.0.x (Last released at May 2018)
>
> Thoughts?
>
> Thanks,
> Wangda



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sean Busbey]

+1

On Tue, Aug 20, 2019 at 10:03 PM Wangda Tan  wrote:
>
> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Sean Busbey]

+1 (non-binding)

On Fri, Aug 23, 2019 at 9:06 PM Wangda Tan  wrote:
>
> Hi devs,
>
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
>
> This voting thread will run for 7 days and will end at Aug 30th.
>
> Please let me know if you have any questions.
>
> Thanks,
> Wangda Tan



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sean Busbey]

On Mon, Aug 26, 2019 at 9:20 AM Eric Badger  wrote:
>
> - Stuff has been going into branch-2 sporadically but I don't who is
> actively
> using that code other than as part of a cherrypick backwards strategy.
>
> - Should we do a 2.10.x release? Or just say "time to upgrade?"
>
> We have talked at a few different Hadoop contributors meetups and community
> syncs and agreed that 2.10 should serve as a "bridge" release for 3.x so
> that 2.x clients can talk to 3.x servers and vice versa without
> compatibility issues. At the last meeting where we discussed this it seemed
> that there were 3 major compatibility issues. The biggest one was in the
> edit logs.
>


Just a quick friendly reminder to everyone that stuff has to make it
to the mailing lists for Hadoop for it to be considered a discussion
that happened "in the community". Especially decisions and the
discussions that lead to them have to happen on our mailing lists and
not at in person events.

-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] GitHub PRs without JIRA number

[Sean Busbey]

We should add a Pull Request Template that specifically calls out the
expectation that folks need to have a JIRA associated with their PR
for it to get reviewed. Expectations around time to response and how
to go about getting attention when things lag would also be good to
include. (e.g. are folks expected to ping on the jira? are folks
expected to email a relevant *-dev list?)

If anyone is interested in doing the work to make it so "test this" /
"retest this" / etc work, open a jira and I'll give you some pointers
of examples to go off of. We use a plugin to do this for yetus based
tests in some HBase repos.

On Wed, Sep 4, 2019 at 1:59 PM Wei-Chiu Chuang
 wrote:
>
> +general@
>
>
> On Wed, Aug 28, 2019 at 6:42 AM Wei-Chiu Chuang  wrote:
>
> > I don't think our GitHub integration supports those commands. Ozone has
> > its own github integration that can test individual PRs though.
> >
> >
> >
> > On Tue, Aug 27, 2019 at 12:40 PM Iñigo Goiri  wrote:
> >
> >> I wouldn't go for #3 and always require a JIRA for a PR.
> >>
> >> In general, I think we should state the best practices for using GitHub
> >> PRs.
> >> There were some guidelines but they were kind of open
> >> For example, adding always a link to the JIRA to the description.
> >> I think PRs can have a template as a start.
> >>
> >> The other thing I would do is to disable the automatic Jenkins trigger.
> >> I've seen the "retest this" and others:
> >> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
> >> https://github.com/jenkinsci/ghprb-plugin/blob/master/README.md
> >>
> >>
> >>
> >> On Tue, Aug 27, 2019 at 10:47 AM Wei-Chiu Chuang 
> >> wrote:
> >>
> >> > Hi,
> >> > There are hundreds of GitHub PRs pending review. Many of them just sit
> >> > there wasting Jenkins resources.
> >> >
> >> > I suggest:
> >> > (1) close PRs that went stale (i.e. doesn't compile). Or even close PRs
> >> > that hasn't been reviewed for more than a year.
> >> > (1) close PRs that doesn't have a JIRA number. No one is going to
> >> review a
> >> > big PR that doesn't have a JIRA anyway.
> >> > (2) For PRs without JIRA number, file JIRAs for the PR on behalf of the
> >> > reporter.
> >> > (3) For typo fixes, merge the PRs directly without a JIRA. IMO, this is
> >> the
> >> > best use of GitHub PR.
> >> >
> >> > Thoughts?
> >> >
> >>
> >



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Prefer Github PR Integration over patch in JIRA

[Sean Busbey]

a word of caution. according to INFRA-18748, asf infra is going to be
cutting out blind PR building. So we'll need to be sure that precommit
integration works e.g. testing PRs because there's a JIRA that a
whitelisted contributor has associated and put in patch available
status.

On Mon, Jul 22, 2019 at 1:06 PM Wei-Chiu Chuang  wrote:
>
> Historically, Hadoop developers create patches and attache them to JIRA,
> andthen the Yetus bot runs precommit against the patch in the JIRA.
>
> The Github PR is more convenient for code review and less hassle for
> committers to merge a commit. I am proposing for the community to prefer
> Github PR over the traditional patch-in-jira. This doesn't mean we will
> reject the traditional way, but we can move gradually to the new way.
> Additionally, update the Hadoop "How to contribute" wiki, and advertise
> that Github PR is the preferred method.
>
> Thoughts?



-- 
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: How should we do about dependency update?

[Sean Busbey]

speaking with my HBase hat on instead of my Hadoop hat, when the
Hadoop project publishes that there's a CVE but does not include a
maintenance release that mitigates it for a given minor release line,
we assume that means the Hadoop project is saying that release line is
EOM and should be abandoned.

I don't know if that's an accurate interpretation in all cases.

With my Hadoop hat on, I think downstream projects should use the
interfaces we say are safe to use and those interfaces should not
include dependencies where practical. I don't know how often a CVE
comes along for things like our logging API dependency, for example.
But downstream folks should definitely not rely on dependencies we use
for internal service, so I'm surprised that a version change for Jetty
would impact downstream.


On Mon, Oct 21, 2019 at 12:33 PM Wei-Chiu Chuang  wrote:
>
> Hi Hadoop developers,
>
> I've always had this question and I don't know the answer.
>
> For the last few months I finally spent time to deal with the vulnerability
> reports from our internal dependency check tools.
>
> Say in HADOOP-16152 
> we update Jetty from 9.3.27 to 9.4.20 because of CVE-2019-16869, should I
> cherrypick the fix into all lower releases?
> This is not a trivial change, and it breaks downstreams like Tez. On the
> other hand, it doesn't seem reasonable if I put this fix only in trunk, and
> left older releases vulnerable. What's the expectation of downstream
> applications w.r.t breaking compatibility vs fixing security issues?
>
> Thoughts?



--
busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] check style changes

[Sean Busbey]

Hi folks!

I’d like to start cleaning up our nightly tests. As a bit of low hanging fruit 
I’d like to alter some of our check style rules to match what I think we’ve 
been doing in the community. How would folks prefer I make sure we have 
consensus on such changes?

As an example, our last nightly run had ~81k check style violations (it’s a big 
number but it’s not that bad given the size of the repo) and roughly 16% of 
those were for line lengths in excess of 80 characters but <= 100 characters.

If I wanted to change our line length check to be 100 characters rather than 
the default of 80, would folks rather I have a DISCUSS thread first? Or would 
they rather a Jira + PR with the discussion of the merits happening there?

—
busbey



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] which release lines should we still consider actively maintained?

[Sean Busbey]

Hi folks!

Which release lines do we as a community still consider actively maintained?

I found an earlier discussion[1] where we had consensus to consider branches 
that don’t get maintenance releases on a regular basis end-of-life for 
practical purposes. The result of that discussion was written up in our wiki 
docs in the “EOL Release Branches” page, summarized here

>  If no volunteer to do a maintenance release in a short to mid-term (like 3 
> months to 1 or 1.5 year).

Looking at release lines that are still on our download page[3]:

* Hadoop 2.10.z - last release 8 months ago
* Hadoop 3.1.z - last release 9.5 months ago
* Hadoop 3.2.z - last release 4.5 months ago
* Hadoop 3.3.z - last release 10 months ago

And then trunk holds 3.4 which hasn’t had a release since the branch-3.3 fork 
~14 months ago.

I can see that Wei-Chiu has been actively working on getting the 3.3.1 release 
out[4] (thanks Wei-Chiu!) but I do not see anything similar for the other 
release lines.

We also have pages on the wiki for our project roadmap of release[5], but it 
seems out of date since it lists in progress releases that have happened or 
branches we have announced as end of life, i.e. 2.8.

We also have a group of pages (sorry, I’m not sure what the confluence jargon 
is for this) for “hadoop active release lines”[6] but this list has 2.8, 2.9, 
3.0, 3.1, and 3.3. So several declared end of life lines and no 2.10 or 3.2 
despite those being our release lines with the most recent releases.

Are there folks willing to go through being release managers to get more of 
these release lines on a steady cadence?

If I were to take up maintenance release for one of them which should it be?

Should we declare to our downstream users that some of these lines aren’t going 
to get more releases?

Is there downstream facing documentation somewhere that I missed for setting 
expectations about our release cadence and actively maintained branches?

Do we have a backlog of work written up that could make the release process 
easier for our release managers?


[1]: https://s.apache.org/7c8jt
[2]: https://s.apache.org/4no96
[3]: https://hadoop.apache.org/releases.html
[4]: https://s.apache.org/1bvwe
[5]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
[6]: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hello!

What do folks think about changing our line length guidelines to allow for 100 
character width?

Currently, we tell folks to follow the sun style guide with some exception 
unrelated to line length. That guide says width of 80 is the standard and our 
current check style rules act as enforcement.

Looking at the current trunk codebase our nightly build shows a total of ~15k 
line length violations; it’s about 18% of identified checkstyle issues.

The vast majority of those line length violations are <= 100 characters long. 
100 characters happens to be the length for the Google Java Style Guide, 
another commonly adopted style guide for java projects, so I suspect these 
longer lines leaking past the checkstyle precommit warning might be a 
reflection of committers working across multiple java codebases.

I don’t feel strongly about lines being longer, but I would like to move 
towards more consistent style enforcement as a project. Updating our project 
guidance to allow for 100 character lines would reduce the likelihood that 
folks bringing in new contributions need a precommit test cycle to get the 
formatting correct.

Does anyone feel strongly about keeping the line length limit at 80 characters?

Does anyone feel strongly about contributions coming in that clear up line 
length violations?


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hi folks!

The consensus seems pretty strongly in favor of increasing the line length 
limit. Do folks still want to see a formal VOTE thread?


> On May 19, 2021, at 4:22 PM, Sean Busbey  wrote:
> 
> Hello!
> 
> What do folks think about changing our line length guidelines to allow for 
> 100 character width?
> 
> Currently, we tell folks to follow the sun style guide with some exception 
> unrelated to line length. That guide says width of 80 is the standard and our 
> current check style rules act as enforcement.
> 
> Looking at the current trunk codebase our nightly build shows a total of ~15k 
> line length violations; it’s about 18% of identified checkstyle issues.
> 
> The vast majority of those line length violations are <= 100 characters long. 
> 100 characters happens to be the length for the Google Java Style Guide, 
> another commonly adopted style guide for java projects, so I suspect these 
> longer lines leaking past the checkstyle precommit warning might be a 
> reflection of committers working across multiple java codebases.
> 
> I don’t feel strongly about lines being longer, but I would like to move 
> towards more consistent style enforcement as a project. Updating our project 
> guidance to allow for 100 character lines would reduce the likelihood that 
> folks bringing in new contributions need a precommit test cycle to get the 
> formatting correct.
> 
> Does anyone feel strongly about keeping the line length limit at 80 
> characters?
> 
> Does anyone feel strongly about contributions coming in that clear up line 
> length violations?
> 

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Sean Busbey]

Hi Bhavik!

What concerns do you have about back porting patches to earlier release 
branches?

If we change our style guidelines then presumably we can do that for all 
branches, so a backport from e.g. trunk to branch-3.3 won’t fail a style check 
on the destination branch unless something changed in the backporting.

If you are referring to patches for clearing up line length violations, my 
usual preference is to aim for my changes to be on all active release lines. So 
at least in the case of the patches coming from me or being committed by me, 
there’d be effort to make sure all branches end up as easy to backport to as 
they were prior to the clean up.



> On May 20, 2021, at 2:27 AM, Bhavik Patel  wrote:
> 
> I am just worried about the backporting of the Jira to child branch!! How
> we are planning to handle this?
> 
> On Thu, May 20, 2021, 11:09 AM Qi Zhu <821684...@qq.com 
> <mailto:821684...@qq.com>> wrote:
> 
>> +1 100 is reasonable.
>> 
>> 
>> 
>> ---Original---
>> From: "Xiaoqiao He"mailto:hexiaoq...@apache.org>
>> Date: Thu, May 20, 2021 13:35 PM
>> To: "Masatake Iwasaki"> <mailto:iwasak...@oss.nttdata.co.jp>;
>> Cc: "Akira Ajisaka"> <mailto:aajis...@apache.org>;"Hadoop Common"<
>> common-...@hadoop.apache.org 
>> <mailto:common-...@hadoop.apache.org>;"Hdfs-dev">  <mailto:hdfs-...@hadoop.apache.org>
>> ;"yarn-dev"> <mailto:yarn-...@hadoop.apache.org>;"mapreduce-dev"<
>> mapreduce-dev@hadoop.apache.org <mailto:mapreduce-dev@hadoop.apache.org>;
>> Subject: Re: [DISCUSS] Change project style guidelines to allow line
>> length 100
>> 
>> 
>> +1 for <= 100 chars long per line length.
>> 
>> On Thu, May 20, 2021 at 10:28 AM Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp wrote:
>> 
>>  I'm +1 too.
>>  I feel 80 characters limit tends to degrade readability by introducing
>>  useless line breaks.
>> 
>>  
>> 
>> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>> 
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>>
>> ;
>>  I have no inconvenience on 100 characters for using Emacs and
>> side-by-side
>>  diff even on 13-inch MBP.
>> 
>>  Masatake Iwasaki
>> 
>>  On 2021/05/20 11:00, Akira Ajisaka wrote:
>>   I'm +1 to allow <= 100 chars.
>>  
>>   FYI: There were some discussions long before:
>>   -
>> 
>> https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>
>> 
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/7813c2f8a49b1d1e7655dad180f2d915a280b2f4d562cfe981e1dd4e%401406489966%40%3Ccommon-dev.hadoop.apache.org%3E>>;
>>  -
>> 
>> https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E>
>> 
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E
>>  
>> <https://lists.apache.org/thread.html/3e1785cbbe14dcab9bb970fa0f534811cfe00795a8cd1100580f27dc%401430849118%40%3Ccommon-dev.hadoop.apache.org%3E>>;
>> 
>>   Thanks,
>>   Akira
>>  
>>   On Thu, May 20, 2021 at 6:36 AM Sean Busbey
>> mailto:sbus...@apple.com.invalid>
>>  wrote:
>>  
>>   Hello!
>>  
>>   What do folks think about changing our line length
>> guidelines to allow
>>  for 100 character width?
>>  
>>   Currently, we tell folks to follow the 

Re: [VOTE] Hadoop 3.1.x EOL

[Sean Busbey]

+1

> On Jun 3, 2021, at 1:14 AM, Akira Ajisaka  wrote:
> 
> Dear Hadoop developers,
> 
> Given the feedback from the discussion thread [1], I'd like to start
> an official vote
> thread for the community to vote and start the 3.1 EOL process.
> 
> What this entails:
> 
> (1) an official announcement that no further regular Hadoop 3.1.x releases
> will be made after 3.1.4.
> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
> 
> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
> 
> Committers are eligible to cast binding votes. Non-committers are welcomed
> to cast non-binding votes.
> 
> Here is my vote, +1
> 
> [1] https://s.apache.org/w9ilb
> [2] 
> https://www.timeanddate.com/worldclock/fixedtime.html?msg=4=20210610T16=248
> 
> Regards,
> Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Sean Busbey]

Sounds good to me. That would be until Thursday June 10th, right?

As a side note it’s concerning that a double-dot maintenance release is a big 
release, but I get that it’s the current state of the project.

> On Jun 3, 2021, at 11:30 AM, Wei-Chiu Chuang  wrote:
> 
> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
> 
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
> 
> Looks like we may need some more time to test. How about extending it by a
> week?
> 
> On Tue, Jun 1, 2021 at 6:29 PM Wei-Chiu Chuang  wrote:
> 
>> Hi community,
>> 
>> This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All blocker
>> issues have been resolved [1] again.
>> 
>> There are 2 additional issues resolved for RC3:
>> * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
>> HADOOP-16878
>> * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source
>> and destination are the same
>> 
>> There are 4 issues resolved for RC2:
>> * HADOOP-17666. Update LICENSE for 3.3.1
>> * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
>> * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
>> * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
>> 
>> The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
>> fixes compared to hadoop-thirdparty 1.1.0:
>> * HADOOP-17707. Remove jaeger document from site index.
>> * HADOOP-17730. Add back error_prone
>> 
>> *RC tag is release-3.3.1-RC3
>> https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
>> 
>> *The RC3 artifacts are at*:
>> https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
>> ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
>> 
>> *The maven artifacts are hosted here:*
>> https://repository.apache.org/content/repositories/orgapachehadoop-1320/
>> 
>> *My public key is available here:*
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> 
>> Things I've verified:
>> * all blocker issues targeting 3.3.1 have been resolved.
>> * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
>> * LICENSE and NOTICE files checked
>> * RELEASENOTES and CHANGELOG
>> * rat check passed.
>> * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
>> * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
>> * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
>> Had to patch some of them due to commons-lang migration (Hadoop 3.2.0) and
>> dependency divergence. Issues are being identified but so far nothing
>> blocker for Hadoop itself.
>> 
>> Please try the release and vote. The vote will run for 5 days.
>> 
>> My +1 to start,
>> 
>> [1] https://issues.apache.org/jira/issues/?filter=12350491
>> [2]
>> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>> 
>> 
>> 



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker

[Sean Busbey]

Hi Brahma!

Thanks for organizing this. What’s the timezone for the 10p - midnight? Pacific 
Time?

> On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula  wrote:
> 
> Hi All,
> 
> Updated the meeting to record the session.. Please use the following link
> to attend the conference tomorrow.
> 
> 
> Uber's story on running Apache Hadoop deployment in Docker
> September 9, 2021, 10:00pm – September 10, 2021, 12:00am ·
> Google Meet joining info
> *Video call link: https://meet.google.com/few-wppc-xoa
> *
> Or dial: ‪(US) +1 443-424-3811‬ PIN: ‪384 713 518‬#
> More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915
> 
> On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula 
> wrote:
> 
>> Hi All,
>> 
>> Happy to announce the following conference. Block your calendar and make
>> yourself available.
>> Thanks Mithun and team for accepting.
>> 
>> 
>> *Topic:*Uber's story on running Apache Hadoop deployment in Docker
>> 
>> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time*
>> 
>> *Meeting Link :   *Google Meet joining info
>>   Video call link:
>> https://meet.google.com/akk-qmzy-qsu
>> 
>> *  Note: *Created Teams meeting also as a backup
>> [2].
>> 
>> *High level Agenda: *
>> 
>>   - Importance of Hadoop Cluster Management (3-5 min)
>>   - Introduction about problem space
>>   - Challenges in this space and why it is important to solve them
>>   - Evolution of Uber Hadoop Cluster Management (10-15 min)
>>   -  How our strategy evolved over time since Hadoop was set up at Uber
>>   -  Key learnings from the evolution over the years
>>   - Our current approach today (15-20min)
>>   -  Key learnings from this major overhaul
>>   -  Current challenges that we are working on
>>   - Q (20 min)
>> 
>> 
>> Aiming for 20 min Q assuming that there will be more questions based on
>> the blog that was published(1) + presentation at the session.
>> 
>> 
>> 
>> 
>> ---
>> 1) https://eng.uber.com/hadoop-container-blog/
>> 
>> 2)
>> 
>> *Join on your computer or mobile app*
>> 
>> Click here to join the meeting
>> 
>> 
>> *Join with a video conferencing device*
>> 
>> 967904...@t.plcm.vc
>> 
>> Video Conference ID: 117 464 626 6
>> 
>> Alternate VTC instructions
>> 
>> 
>> *Or call in (audio only)*
>> 
>> +1 213-204-8714,,273569658# <+12132048714,,273569658#>   United States,
>> Los Angeles
>> 
>> (833) 827-4491,,273569658# <8338274491,,273569658#>   United States
>> (Toll-free)
>> 
>> Phone Conference ID: 273 569 658#
>> 
>> Find a local number
>> 
>> | Reset PIN 
>> 
>> 
>> 
>> -- Brahma Reddy Battula
>> 
> 
> 
> -- 
> 
> 
> 
> --Brahma Reddy Battula



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Migrate to Yetus Interface classification annotations

[Sean Busbey]

I think consolidating on a common library and tooling for defining API 
expectations for Hadoop would be great.

Unfortunately, the Apache Yetus community recently started a discussion around 
dropping their maintenance of the audience annotations codebase[1] due to lack 
of community interest. In particular, there has been an outstanding problem 
with doclet support for filtering javadocs by annotation since JDK9 came out.

I think that means a necessary first step here would be to determine if we have 
contributors willing to show up over in that project to get things into a good 
state for future JDK adoption.



[1]:
https://s.apache.org/ybdl6
"[DISCUSS] Drop JDK8; audience-annotations" from d...@yetus.apache.org

> On Sep 27, 2021, at 2:46 AM, Viraj Jasani  wrote:
> 
> Since the early days, Hadoop has provided Interface classification
> annotations to represent the scope and stability for downstream
> applications to select Hadoop APIs carefully. After some time, these
> annotations (InterfaceAudience and InterfaceStability) have been migrated
> to Apache Yetus. As of today, with increasing number of Hadoop ecosystem
> applications using (or starting to use) Yetus stability annotations for
> their own downstreamers, we should also consider using IA/IS annotations
> provided by *org.apache.yetus.audience *directly in our codebase and retire
> our *org.apache.hadoop.classification* package for the better separation of
> concern and single source.
> 
> I believe we can go with this migration to maintain compatibility for
> Hadoop downstreamers:
> 
>   1. In Hadoop trunk (3.4.0+ releases), replace all usages of o.a.h.c
>   stability annotations with o.a.y.a annotations.
>   2. Deprecate o.a.h.c annotations, and provide deprecation warning that
>   we will remove o.a.h.c in 4.0.0 (or 5.0.0) release and the only source for
>   these annotations should be o.a.y.a.
> 
> Any thoughts?



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging PRs vs. commit from CLI and keep committer field

[Sean Busbey]

If you add a line in the commit message that the commit closes a given PR # 
then GitHub will annotate the PR as related to the specific commit and close it 
for you.

i.e. you can add “closes #3454” to the commit message body and then PR 3454 
will close and link to that commit when it hits the default branch.

I believe these docs describing associating a GitHub issue via a commit message 
also apply to associating commits to pull requests, if you are interested in 
what specific phrases work:

https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword


Has anyone tried handling a squash merge via the GitHub CLI tooling rather than 
the web or plain git tooling? Does it similarly overwrite committer metadata?

e.g. the GitHub CLI example from the merging PRs docs?

https://docs.github.com/en/github/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request#merging-a-pull-request
 




> On Oct 25, 2021, at 9:57 AM, Szilárd Németh  wrote:
> 
> Hi Ayush,
> Thanks a lot for your response.
> 
> 
>> Yahh, remember chasing Github support regarding this last year and they
>> reverted back that they are aware of this and have an internal jira for
>> this but can not comment upon how much time it would take. (As of now 1
>> year & counting)
>> 
> So if I get this right: Their internal jira is in the same untouched state
> and they wouldn't make any progress?
> 
> 
> Just in case you want to use the CLI & still make the PR marked as merged,
>> A dirty way to do this is:
>> Pull the changes to your local.
>> Rebase & Squash all commits, in the commit message in the end put actual
>> the PR number, eg. #1234,
>> force push to the PR, that is the contributors fork, this will update his
>> PR and then merge the Branch to trunk in your local and push. It marks
>> the PR as merged, at least last year it did when I tried. :-)
>> 
> 
> Thanks for this dirty hack :)
> Probably I will refrain from doing this, I don't like to force push if it's
> not "mandatory".
> Anyway, if the community is fine with it, I will continue to commit from
> the CLI and close the PR, even though it's not that convenient.
> 
> 
> Best,
> Szilard
> 
> 
> 
> On Wed, 20 Oct 2021 at 05:07, Ayush Saxena  wrote:
> 
>> Yahh, remember chasing Github support regarding this last year and they
>> reverted back that they are aware of this and have an internal jira for
>> this but can not comment upon how much time it would take. (As of now 1
>> year & counting)
>> 
>> As far as I remember this is with squash & commit only. If you do say a
>> rebase & merge. The commit email id is preserved. But other options we have
>> blocked.
>> 
>> I think most of the projects are using squash & merge and many people
>> won’t be ok to use CLI rather than UI
>> So, I don’t think we have an ALT here.
>> 
>> Just in case you want to use the CLI & still make the PR marked as merged,
>> A dirty way to do this is:
>> Pull the changes to your local.
>> Rebase & Squash all commits, in the commit message in the end put actual
>> the PR number, eg. #1234,
>> force push to the PR, that is the contributors fork, this will update his
>> PR and then merge the Branch to trunk in your local and push. It marks the
>> PR as merged, at least last year it did when I tried. :-)
>> 
>> -Ayush
>> 
>> 
>>> On 20-Oct-2021, at 3:27 AM, Szilárd Németh  wrote:
>>> 
>>> Hi,
>>> 
>>> I noticed something strange in our commits, in particular the committer
>>> field is not reflecting the user who committed the commit.
>>> 
>>> *1. First, I wanted to check Gergely's commits from the last month or so.
>>> This was getting to be suspicious as I expected to see a bunch of commits
>>> from Sept / Oct of this year. *
>>> 
>>> *git log CLI output:*
>>> ➜ git --no-pager log --format=fuller --committer=shuzirra
>>> commit 44bab51be44e31224dabbfa548eb27ea5fb2f916
>>> Author: Gergely Pollak 
>>> AuthorDate: Wed Aug 4 15:43:07 2021 +0200
>>> Commit: Gergely Pollak 
>>> CommitDate: Wed Aug 4 15:43:57 2021 +0200
>>> 
>>> 
>>>   YARN-10849 Clarify testcase documentation for
>>> TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by
>>> Szilard Nemeth
>>> 
>>> 
>>> commit e9339aa3761295fe65bb786e01938c7c177cd6e7
>>> Author: Gergely Pollak 
>>> AuthorDate: Tue Jun 1 15:57:22 2021 +0200
>>> Commit: Gergely Pollak 
>>> CommitDate: Tue Jun 1 15:57:22 2021 +0200
>>> 
>>> 
>>>   YARN-10797. Logging parameter issues in scheduler package. Contributed
>>> by Szilard Nemeth
>>> 
>>> 
>>> *2. Another example of a merged PR, here I was the author and Adam Antal
>>> was the committer:  *
>>> PR link: https://github.com/apache/hadoop/pull/3454
>>> 
>>> *git log CLI output:*
>>> ➜ git --no-pager log --format=fuller